Skill: wbavon/perf-test-flagos (version 1.0.0)

--- name: perf-test-flagos description: | Run accuracy benchmarks (FlagEval, when available) and performance benchmarks (vllm bench serve) against a served model. Covers 5 workload profiles: short/long prefill x short/long decode + high concurrency. Collects throughput, latency, TTFT, TPOT metrics. user-invokable: true allowed-tools: "Bash(*) Read Edit Write Glob Grep WebSearch WebFetch AskUserQuestion" --- # Accuracy + Performance Test Start vLLM serve with the target model, run accuracy benchmarks (when FlagEval is available) and performance benchmarks (vllm bench serve) across multiple profiles. ## Skill Components ``` perf-test/ ├── SKILL.md # This file — execution flow ├── scripts/ │ ├── run_benchmark.py # Run single benchmark profile (JSON output) │ └── run_all_benchmarks.py # Run all 5 profiles, collect + summarize (JSON) └── references/ └── benchmark-profiles.md # Profile definitions, metrics, vllm bench usage ``` **Reused from env-verify:** - `env-verify/scripts/test_serve_mode.py` — can be used to verify server is healthy before benchmarking (optional pre-check) ## Prerequisites - Running container with software stack installed - model-verify completed — know which stack to use (`full` vs `base`) - Model path, TP size, and recommended stack from model-verify If invoked standalone, ask for container name, model path, TP size, and stack config. If invoked from `/flagrelease`, these are passed as context. ## Execution Flow ### Step 1: Start vLLM Server Use the stack recommended by model-verify. Read `references/benchmark-profiles.md` for the vllm serve command pattern. ```bash docker exec -d <CONTAINER> bash -c ' export USE_FLAGGEMS=<0|1> export FLAGCX_PATH=<path_or_unset> export VLLM_PLUGINS=<fl_or_unset> vllm serve <MODEL_PATH> \ --tensor-parallel-size <TP_SIZE> \ --max-num-batched-tokens 4096 \ --max-num-seqs 256 \ --trust-remote-code \ --port 8000 \ <EXTRA_ARGS> ' ``` Wait for server ready (poll /health, timeout 300s): ```bash docker exec <CONTAINER> bash -c ' for i in $(seq 1 150); do if curl -s http://localhost:8000/health 2>/dev/null | grep -qE "ok|200|\{\}"; then echo "SERVER_READY"; break fi sleep 2 done ' ``` If server doesn't start, report error and exit. ### Step 2: Get Model Name from Server ```bash docker exec <CONTAINER> bash -c ' curl -s http://localhost:8000/v1/models | python3 -c " import json, sys; print(json.load(sys.stdin)[\"data\"][0][\"id\"])" ' ``` --- ## Part A: Accuracy Test (FlagEval) — PLACEHOLDER **STATUS:** FlagEval test client not yet available. When FlagEval becomes available, update this section with: - [ ] Docker image URL or pip package name - [ ] Supported benchmarks (MMLU, GSM8K, HumanEval, etc.) - [ ] Required arguments and configuration - [ ] Expected output format - [ ] Pass/fail criteria (accuracy thresholds) **Current behavior:** Report accuracy test as SKIPPED. --- ## Part B: Performance Benchmarks ### Step 3: Run All Benchmark Profiles Copy scripts into the container and run: ```bash docker cp <SKILL_DIR>/scripts/run_benchmark.py <CONTAINER>:/tmp/ docker cp <SKILL_DIR>/scripts/run_all_benchmarks.py <CONTAINER>:/tmp/ docker exec <CONTAINER> python3 /tmp/run_all_benchmarks.py \ --model <MODEL_NAME> \ --tokenizer <MODEL_PATH> \ --port 8000 \ --output-dir /data/results/perf ``` The script runs all 5 default profiles (see `references/benchmark-profiles.md`), saves per-profile JSON to `/data/results/perf/`, and outputs a combined JSON report with a summary table. **Important:** One profile failure does NOT skip remaining profiles. ### Step 4: Stop Server ```bash docker exec <CONTAINER> bash -c 'pkill -f "vllm serve" || true' ``` ### Step 5: Produce Report ```json { "status": "PASS | PARTIAL | FAIL", "stage": "perf-test", "model": "<MODEL_PATH>", "tensor_parallel_size": 8, "flags": {"USE_FLAGGEMS": "1|0", "FLAGCX_PATH": "..."}, "accuracy": { "status": "SKIPPED", "reason": "FlagEval test client not yet available" }, "performance": { "status": "PASS | PARTIAL | FAIL", "profiles_passed": "5/5", "profiles": [ "...per-profile results..." ], "summary_table": "...markdown table..." } } ``` Present the summary table to the user: ``` | Profile | Input | Output | Prompts | Req/s | Tok/s | TTFT(ms) | TPOT(ms) | P99(ms) | Status | |---------|-------|--------|---------|-------|-------|----------|----------|---------|--------| | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ``` **Status logic:** - `PASS` — all profiles completed - `PARTIAL` — some passed, some failed - `FAIL` — server didn't start or all profiles failed ## Error Handling | Failure | Behavior | |---------|----------| | Server fails to start | Report error; exit | | `vllm bench serve` not found | Report vllm version issue | | Single profile fails | Report error, continue remaining profiles | | Single profile times out | Kill after 600s, report partial, continue | | Server crashes mid-benchmark | Capture logs, report which profile caused crash | | OOM during high concurrency | Report, suggest reducing num_prompts | ## Timeout Rules | Operation | Timeout | |-----------|---------| | Server startup | 300s | | Per profile benchmark | 600s |


Security scan verdicts — ClawScan: suspicious | static analysis: clean | VirusTotal:  | SkillSpector: clean (LOW)