EgoPQ-project/EgoPQ-caches

EgoPQ Real-Serving Handoff

This is the single entry point for transferring the native W4A8/KV8 serving work. The handoff has three independent parts: source code, external model artifacts, and validation evidence. A Git branch alone is not a complete handoff because the calibration caches and packed checkpoints are too large for Git.

For a command-by-command run, start with docs/real_serving_quickstart.md. This document records artifact provenance, validation order, and the remaining model-family boundary.

1. Source Checkpoint

Use the exact branch and record its commit before running anything:

git clone --branch exp/fused-serving-stack --single-branch \
  https://github.com/think7z/eoc.git egopq
cd egopq
git rev-parse HEAD

The handoff commit must include this document, the consolidated OmniServe v4 patch, and scripts/bootstrap_omniserve_eoc.sh. Do not use the old disposable checkout at /scratch/gsun20/eoc/third_party/omniserve; its Git objects and implementation files were removed by scratch cleanup.

Rebuild the patched dependency in a new directory:

bash scripts/bootstrap_omniserve_eoc.sh \
  /scratch/$USER/eoc/third_party/omniserve-eoc

This clones OmniServe commit 02b2925aa6fa3b92b06316a1524b7f38922cd9c8 and applies the reviewed patches/omniserve_eoc_native_vlm_serving_v4.patch. The patch contains the native VLM request bridge, packed QuaRot model, fine-grained KV fixes, prompt compaction, and the optional CUTLASS large-prefill dispatch. The earlier patch files are development history and are not part of the reproduction sequence.

Build the dependency and the repository-local kernels using the commands in e2e/real_serving/README.md and models/VideoLlama3/real_kernels/README.md.

2. External Artifacts

There are two cache classes and they are not interchangeable.

Purpose	VL3 artifact	Required property
Paper fake-quant accuracy	videollama3_gptq_w4_asym_clip_calvidmixed_vr50_f8.pt.preseal_phase5	Historical paper cache; decoder scale/zero buffers are absent.
Native packed W4A8/KV8	videollama3_gptq_w4_asym_clip_calvidmixed_vr50_f8_realcal.pt	Exact _gptq_scale and _gptq_zero metadata.

On SOL, the authoritative files are under /scratch/gsun20/eoc/quarot_cache. Collaborators need read access to those files or immutable copies in their own artifact directory. Do not rename a realcal cache to the preseal filename or overwrite either artifact.

Verify paper caches:

bash scripts/verify_preseal_caches.sh --sha256

Native exact-metadata cache identities:

VL3
  bytes:  16382920401
  sha256: 8512b050fa05dfa765f35a568af31b09812b23d722c002900f42e9f2be8038da

Qwen2.5-VL
  bytes:  16596682455
  sha256: ad6ce094dcd9d4daae64a8300a8dd399675332c9cb75f6171c245d17b2142926

Verify those identities with:

bash scripts/verify_realcal_caches.sh --sha256

For an off-cluster handoff, publish these immutable files in Hugging Face, Zenodo, or another versioned artifact store and record the download URI next to the checksums. GitHub is only the source distribution.

The private artifact mirror is EgoPQ-project/EgoPQ-caches. Request repository access from the owner, authenticate with hf auth login, and download the complete immutable bundle with:

HF_HOME=/scratch/$USER/huggingface \
~/.conda/envs/eocbench/bin/hf download EgoPQ-project/EgoPQ-caches \
  --repo-type model \
  --local-dir /scratch/$USER/eoc/egopq_caches

cd /scratch/$USER/eoc/egopq_caches
sha256sum -c SHA256SUMS

The repository is intentionally private because these caches contain derived model weights. Do not make it public until the upstream model licenses and distribution conditions have been reviewed.

Packed checkpoints are derived artifacts. Re-export them from the verified realcal cache instead of relying on a scratch copy:

OMNISERVE_ROOT=/scratch/$USER/eoc/third_party/omniserve-eoc
CACHE_BUNDLE=/scratch/$USER/eoc/egopq_caches
CUDA_VISIBLE_DEVICES=0 \
PYTHONPATH=${OMNISERVE_ROOT}:${OMNISERVE_ROOT}/kernels:${PYTHONPATH:-} \
~/.conda/envs/eocbench/bin/python scripts/export_omniserve_packed_qserve.py \
  --cache-path ${CACHE_BUNDLE}/realcal/vl3/videollama3_gptq_w4_asym_clip_calvidmixed_vr50_f8_realcal.pt \
  --output-dir /scratch/$USER/eoc/omniserve_packed/vl3_calvidmixed_realcal_w4a8kv8_qserve_quarotv2 \
  --device cuda

The exporter must report 28 layers, 196 packed linears, and zero skipped linears. Its eoc_packed_metadata.json is part of the artifact provenance.

3. Validation Order

Run checks in this order so dependency, packing, and model errors remain separable:

# CPU/source-level checks
~/.conda/envs/eocbench/bin/python -m pytest -q \
  tests/test_real_serving_helpers.py \
  tests/test_omniserve_vlm_bridge.py

# Repository-local CUDA kernels
CUDA_VISIBLE_DEVICES=0 ~/.conda/envs/eocbench/bin/python \
  tests/test_quant4_linear.py

# Packed OmniServe load + prefill/decode smoke
CUDA_VISIBLE_DEVICES=0 \
PYTHONPATH=${OMNISERVE_ROOT}:${OMNISERVE_ROOT}/kernels:${PYTHONPATH:-} \
~/.conda/envs/eocbench/bin/python \
  scripts/e2e_vl3_omniserve_packed_smoke.py \
  --quant-path /scratch/$USER/eoc/omniserve_packed/vl3_calvidmixed_realcal_w4a8kv8_qserve_quarotv2 \
  --seq-len 8 --max-new-tokens 2

Only after these pass should the collaborator run EOC accuracy or matched TTFT profiling. Report the main-repository commit, OmniServe upstream commit, v4 patch checksum, cache checksum, packed metadata, GPU model, and all runtime flags with every result.

Known Boundary

VL3 is the validated native packed path. Qwen packed serving remains a smoke/parity target until its full M-RoPE accuracy path is revalidated. The historical preseal cache is the paper fake-quant accuracy artifact; the realcal cache is the strict native-serving artifact. Mixing those two paths produces plausible-looking but non-comparable numbers.