Instructions to use SamChen567/phase2_qbase_enhance with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use SamChen567/phase2_qbase_enhance with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="SamChen567/phase2_qbase_enhance", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("SamChen567/phase2_qbase_enhance", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use SamChen567/phase2_qbase_enhance with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "SamChen567/phase2_qbase_enhance"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase2_qbase_enhance",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/SamChen567/phase2_qbase_enhance

SGLang

How to use SamChen567/phase2_qbase_enhance with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "SamChen567/phase2_qbase_enhance" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase2_qbase_enhance",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "SamChen567/phase2_qbase_enhance" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "SamChen567/phase2_qbase_enhance",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use SamChen567/phase2_qbase_enhance with Docker Model Runner:
```
docker model run hf.co/SamChen567/phase2_qbase_enhance
```

phase2_qbase_enhance (native-res, video + images, 1 epoch)

Phase 2 of the qbase video-token compressor curriculum, built on VideoLLaMA3-7B (DAMO-NLP-SG/VideoLLaMA3-7B). It continues training the learned-query compressor ("qbase") that sits between the frozen SigLIP-NaViT vision encoder and the frozen Qwen2-7B LLM. The training signal is still answer cross-entropy, but the data is now a broad video-and-image SFT mix instead of captions only.

Warm start: SamChen567/phase1_qbase_internvid (qbase + projector + the two compression-token embedding rows).

This checkpoint needs the custom code in the training repository (videollama3/model, Videollama3Qwen2ForCausalLM with token_compressor). It does not load with stock transformers alone.

What is trained

module	status
vision encoder (SigLIP-NaViT)	frozen
`token_compressor` (qbase, 170M)	trained, LR 2e-5
`mm_projector` (mlp2x_gelu, 17M)	trained, LR 1e-6
`<\|compression_start\|>` / `<\|compression_end\|>` embedding rows	trained (all other rows frozen)
LLM (Qwen2-7B) + `lm_head`	frozen

Compressor

The compressor architecture is the same as in Phase 1.

transformer_decoder_flat: 8 layers, 8 heads, hidden 1152, K = 64 learned queries per segment, with cross-attention onto the segment's encoder tokens using (t, h, w) RoPE.
Adaptive segmentation (not learned): a video gets N = ⌊T/4⌋ + 1 segments. There is a forced cut every 8 frames, and the remaining cuts go where the per-frame mean encoder feature changes most (largest cosine distance between consecutive frames). Each segment is 1–8 frames long. Segmentation is deterministic (segment_sample_tau = 0), and the target frames per segment is fixed at 4 (TF_CHOICES = 4, not mixed 1–4).
A still image is a single segment of 64 tokens.
Option A (match_encoder_scale): the output is rescaled to the encoder tokens' per-dim mean/std, followed by a learnable per-channel γ/β.
Option B (distr_loss_weight = 0.05): a CORAL-style distribution-matching auxiliary loss. The note at the end explains why its effective weight is higher than 0.05.

Input layout seen by the LLM

Each segment carries its own time range (segment_time_marks = True), the same as in Phase 1:

Time:0s-3s:<|compression_start|>[64]<|compression_end|>Time:4s-4s:<|compression_start|>[64]<|compression_end|> ...

Data

phase2_all: 755,269 samples, a reorganised union of the Video-XL / Video-XL-Pro SFT data covering 16 tasks. It includes every still image in that data, so images are 50 % of the samples.
Videos are decoded at 1 fps with max_frames = 180, at native resolution. The per-video budget is 65,536 merged tokens, with at most 2,048 and at least 64 tokens per frame.
Samples are grouped by estimated encoder cost (group_by_encode_cost) to balance the load across GPUs.
14 samples failed to decode during training (empty image lists). Each was replaced with a randomly drawn sample.

Training


epochs / steps	1 / 1,574
global batch	480 (2 per device × 8 GPUs × 30 grad-acc, flattened sequences)
LR	qbase 2e-5, projector 1e-6; 3 % warmup, cosine; weight decay 0
precision / parallel	bf16, DeepSpeed ZeRO-1, gradient checkpointing (LLM + compressor)
wall time	15.1 h (54,210 s)

The exact launcher is included as launcher.sh; this run used its defaults. The full loss curve is in trainer_state.json.

Loss

steps	logged loss (mean)
1–25	0.1711
400–424	0.1590
775–799	0.1567
1200–1224	0.1534
1550–1574 (end)	0.1538

The reported train_loss is 0.1566. The run had no NaN losses, and the gradient norm stayed between 0.075 and 0.94.

trainer_state.json also contains per-micro-batch diagnostics logged on rank 0. These are per-token CE values and are not divided by 8:

steps	`loss_img` (still-image tokens)	`loss_vid` (video tokens)
1–100	1.074	0.931
700–800	0.951	0.864
1475–1574	0.929	0.847

These are training losses only. No held-out evaluation is reported here.

Notes on loss scaling (known quirk of the training code)

The custom compute_loss divides the CE sum by the token count gathered across all 8 GPUs but does not multiply back by the number of processes. As a result, the logged loss is ≈ per-token CE / 8.
The Option-B term is divided only by the number of grad-accum steps, not by 8, so relative to CE its effective weight is ≈ 0.05 × 8 = 0.4.
Phase 1 has the same behaviour.