Instructions to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="BatuhanECB/vibethinker-3b-finance-sft-quantized-v1")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("BatuhanECB/vibethinker-3b-finance-sft-quantized-v1", dtype="auto")

llama-cpp-python

How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="BatuhanECB/vibethinker-3b-finance-sft-quantized-v1",
	filename="gguf/student-Q2_K-imat.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

Use Docker

docker model run hf.co/BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

LM Studio
Jan

vLLM

How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "BatuhanECB/vibethinker-3b-finance-sft-quantized-v1"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BatuhanECB/vibethinker-3b-finance-sft-quantized-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

SGLang

How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "BatuhanECB/vibethinker-3b-finance-sft-quantized-v1" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BatuhanECB/vibethinker-3b-finance-sft-quantized-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "BatuhanECB/vibethinker-3b-finance-sft-quantized-v1" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "BatuhanECB/vibethinker-3b-finance-sft-quantized-v1",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with Ollama:
```
ollama run hf.co/BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M
```

Unsloth Studio

How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 to start chatting

How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

Run Hermes

hermes

Atomic Chat new

OpenClaw new

How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Docker Model Runner
How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with Docker Model Runner:
```
docker model run hf.co/BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M
```

Lemonade

How to use BatuhanECB/vibethinker-3b-finance-sft-quantized-v1 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull BatuhanECB/vibethinker-3b-finance-sft-quantized-v1:Q4_K_M

Run and chat with the model

lemonade run user.vibethinker-3b-finance-sft-quantized-v1-Q4_K_M

List all available models

lemonade list

VibeThinker-3B · Finance-Reader SFT — Quantized Pack

A 3-billion-parameter financial-text reader, SFT-distilled from a 35B reasoning teacher, then quantized into many formats. It reads a piece of financial text and returns strict, schema-valid JSON describing what the text implies ex-ante (before knowing the outcome). It is a reader / structured extractor, not a price predictor — see the honest evaluation below.

This repo ships the bf16 source weights + every quantization that works on commodity (Ampere) GPUs, each verified on the full evaluation suite to be finance-task lossless.

🔗 Links

📊 Training data: BatuhanECB/vibethinker-3b-finance-sftmini-data — the exact teacher-distilled SFT set this model was trained on (private).

💻 Code, full pipeline & reports: github.com/batuhan3526/vibe_train — selection → teacher labeling → SFT → eval → quantization, all reproducible.

1. Why this model exists

This is the 3rd generation of a finance-reasoning LLM project. The thesis: a large, expensive reasoning model can read financial text very well, but you cannot afford to run a 35B model over millions of documents all day. So we distill the 35B teacher's reading ability into a 3B student that:

runs on a single consumer GPU (or CPU via GGUF),
emits 100% schema-valid JSON natively (no grammar/guided decoding needed),
and matches the teacher's reading (~87–88% direction agreement) at a fraction of the cost.

The goal is a cheap, fast, format-native finance reader for bulk document processing — turning raw news / filings / earnings / regulatory text into structured records.

Intended use & origin — a feature extractor for a sentiment pipeline

This model was built for a concrete production need: a medallion (Bronze → Silver → Gold) sentiment-feature pipeline over 16 US ETFs (5 index + 11 GICS sector). That pipeline ingests raw text — news (NewsAPI/Finnhub/RSS), Google Trends, fear&greed, SEC full-text filings (8-K/risk), regulatory/Federal-Register — and must turn each item into structured features for downstream ML and dashboards.

The job is interpretation, not prediction: make sense of each incoming item — what is this? is it market-relevant? what direction / materiality does the text itself imply? — and emit a structured record. A mechanism that predicts price from news is explicitly NOT wanted. That is exactly why this model is a reader / extractor, not a forecaster — and why its at-chance predictive score (§6) is by design, a feature not a bug. It slots into the pipeline's Tier-3 "DEEP / LLM" stage (the *_llm_deep, *_regulatory_llm, *_filing_llm–style steps), served behind a small local vLLM endpoint, feeding the Silver/Gold feature tables.

Why so small, and why all these quants: the production box is a single RTX A2000 Ada · 16 GB · 70 W low-profile GPU that must also host other models at the same time — no room for a 35B teacher or even a 7B. So the target was the smallest model that can still read finance text into clean structured features. Hence a 3B, and especially the 2–4 GB quants (W4A16 / GGUF Q4_K_M, or the imatrix Q3/Q2 at 1.3–1.7 GB) that leave most of the 16 GB free for the rest of the stack. It was trained on a deliberately small dataset and is shared for anyone with the same constraint: a tiny model that understands and structures financial text on a power- and VRAM-limited GPU.

2. What it does — the task & output format

Input: one financial text (news article, SEC filing chunk, earnings line, regulatory notice, …). Output: strict JSON, ex-ante (text-only; the model never sees future prices).

{
  "reasoning": "2-4 concise third-person factual sentences about what the text implies",
  "event_type": "earnings | filing | regulatory | news | metadata | ...",
  "entities": ["only tickers/orgs EXPLICITLY named in the text"],
  "sector_context": ["inferred sectors/themes, e.g. semiconductors"],
  "market_relevance": "none | low | medium | high",
  "direction": "bearish | slightly_bearish | neutral | slightly_bullish | bullish",
  "score_bucket": -1.0 | -0.5 | -0.2 | 0.0 | 0.2 | 0.5 | 1.0,
  "confidence": 0.0-1.0
}

Calibration rules baked into the prompt/training:

score_bucket is graded by materiality: ±1.0 = major catalyst (M&A, big beat+guidance, enforcement, bankruptcy); ±0.5 = clear moderate; ±0.2 = mild/mixed; 0.0 = no market-relevant content (filing headers, administrative/Federal-Register notices, pure metadata).
If market_relevance = none → score_bucket = 0.0 and direction = neutral (enforced).
direction is derived from score_bucket (no sentiment/score contradiction).
entities are in-text only (no fabricated/sector entities); confidence = quality of evidence, not strength of direction.
reasoning is third-person factual — no first person, no deliberation.

3. Training data

📊 Released as a dataset: BatuhanECB/vibethinker-3b-finance-sftmini-data (9,588 train / 411 eval, with the full schema, label distributions, and the realized-return RLVR sidecar documented).

Corpus: a deduped, time-stratified 1GB sample of a financial corpus (~2.23M records, 7 sources): sec_fulltext (SEC filing chunks), news_other (gnews/newsapi/rss), finnhub_news, regulatory (SEC RSS / Federal Register / enforcement), sec_edgar (filing metadata), numeric (election polls, trends, fear&greed, EPU), earnings.

Selection for this run: a deterministic, source-stratified 10,000-record sample (seed 42), favoring rich text — sec_fulltext 2500 · news_other 2500 · finnhub 1500 · regulatory 1500 · numeric 1000 · sec_edgar 500 · earnings 500. After cleaning + teacher labeling → 9,588 train / 411 held-out.

Data hygiene fixes applied (verified):

Earnings ticker disambiguation — bare-ticker earnings ("MS earnings…") made the teacher hallucinate the wrong company (MS→Microsoft, SO→Samsung). Fixed by injecting canonical Company (TICKER), sector: X into the text before the teacher reads it (8,021-ticker map), + a reasoning-repair pass.
No-signal neutrality — administrative/metadata text forced to 0.0 / neutral (no positive-default bias).
Junk filters: fake 1900-era policy-uncertainty dates, literal "None" earnings, pre-1995 dates.

Forward-return sidecar (for future RLVR, NOT used in this SFT): prices for the current S&P 500 (adj_close, 1990→2026), realized T+{5,20,60} excess return vs SPY + sector ETF, point-in-time (future never enters the input; return is label/reward only).

4. How it was trained

Teacher (label generator): Qwen3.6-35B-A3B-AWQ — a hybrid Mamba/attention MoE (256 experts, 8 active), served offline via vLLM with guided JSON (the schema above). The teacher reads each text and writes the ex-ante JSON; quality enforced by schema validators + normalization (no gold labels were available, so the teacher is the reference).

Student (this model): VibeThinker-3B (a Qwen2.5-Coder-3B-architecture model: 36 layers, hidden 2048, GQA, 151,936 vocab, 128K context), full fine-tuned on the teacher-labeled data.

SFT recipe (2×RTX 3090, WSL):

torchrun 2-GPU + DeepSpeed ZeRO-3 + bitsandbytes 8-bit AdamW + bf16 + gradient checkpointing + Liger fused LM-head/CE (to fit the vocab×seq logits at seq 4096).
completion_only_loss (prompt/completion masking), lr 1e-5 cosine, 2 epochs (~736 steps), seq 4096, effective batch 32.
18.5% math/code replay (GSM8K + MBPP) mixed in for forgetting mitigation.
Train loss 2.59 → ~0.21, token-accuracy ~0.93. Power-loss-resumable (checkpoint-by-id).

Two distinct goals motivated the design: (a) a cheap reader — achieved by this SFT; (b) a predictive edge — NOT achievable by SFT alone (see §5/§6); that requires RLVR on the realized-return reward (future work).

5. Evaluation — reading / processing (the model's actual job)

Field-by-field agreement with the 35B teacher on 500 unseen records (teacher = reference):

field	student-3B vs teacher
valid JSON	100% (native, no guided decoding)
direction (3-way bull/bear/neutral)	88%
direction (5-way)	82%
market_relevance (4-class)	80%
event_type (semantic family)	82%
score_bucket (exact)	29% · MAE 0.325 (direction right, magnitude drifts)
entities (Jaccard)	73%
confidence (MAE)	0.179

Per-source 3-way direction agreement: sec_edgar 100% · regulatory 100% · earnings 97% · numeric 96% · finnhub 80% · news_other 75% · **sec_fulltext 69%** (long docs — truncation, fixable by serving at 8192 ctx).

→ A near-teacher-quality finance reader on clear content (96–100%); the only soft spot is very long filings. Base VibeThinker emitted only 1% valid JSON before SFT → the fine-tune taught the whole task.

Worked examples — teacher (35B) vs this student (3B), one per source

On held-out records, the input text → the teacher's JSON → this student's JSON:

source	input (excerpt)	teacher →	student →
earnings	"CME earnings: EPS 1.39 vs est 1.36, surprise 1.9%"	slightly_bullish · 0.2 · med · 0.9	slightly_bullish · 0.2 · med · 0.9 ✅ exact
sec_fulltext	"FORM 8-K … HCA INC … $1.25B senior unsecured notes"	neutral · 0.2 · med · 0.9 (debt issuance)	slightly_bullish · 0.2 · med · 0.9 (≈, dir off by one)
sec_edgar	"PROTECTION ONE ALARM MONITORING INC (CIK…) — 8-K"	neutral · none (filing header)	neutral · 0.0 · none ✅ correct no-signal
finnhub_news	"Accenture Launches AI Adoption Maturity Model w/ Carnegie Mellon"	slightly_bullish · 0.2 · med · 0.8	slightly_bullish · 0.2 · med · 0.8 ✅ match
news_other	"US stocks fall on a shaky Wall Street as Brent oil > $107"	slightly_bearish · −0.2 · med · 0.8	slightly_bearish · −0.2 · med · 0.8 ✅ match
regulatory	"SEC enforcement action: Randy A. Hamdan and Oracle Consultants"	bearish · −0.2 · low · 0.8	bearish · −0.5 · med · 0.9 (dir ✅, stronger)
numeric	"Search interest: crash 4, recession 33, inflation 75, rates 53"	slightly_bearish · −0.2 · low	neutral · 0.0 · none (student more conservative)

Format per field: direction · score_bucket · market_relevance · confidence. Pattern (matches the metrics above): exact agreement on clear single-name events (earnings, news, finnhub), correct no-signal handling on filing headers (sec_edgar), and occasional divergence on ambiguous macro/aggregate text (numeric search-interest), where the 3B student tends to read "no company-specific signal → neutral."

Does it actually reason — or just classify?

The table above shows only the labels; the model also writes a reasoning field, and it contains genuine financial interpretation. The clearest example — an HCA Inc. Form 8-K (debt issuance):

Teacher (35B): "Form 8-K filing by HCA Inc. detailing the issuance of $1.25 billion in senior unsecured notes (5.500% due 2009 and 6.375% due 2015) through an underwriting agreement with J.P. Morgan and Merrill Lynch. The filing explicitly states that over 10% of net proceeds will be used to repay borrowings from underwriter affiliates, requiring a qualified independent underwriter per NASD rules. This represents a standard debt issuance with specific use of proceeds for refinancing, indicating a neutral capital-structure adjustment without major operational catalysts."

Student (3B): *"…$1.25 billion public offering of senior unsecured notes… maturity dates and interest-payment schedules for the 5.500% Notes due 2009 and 6.375% Notes due 2015. This represents a standard debt-issuance event with no immediate earnings or operational-guidance implications."*

There is real financial reasoning here — not just a label. The model:

identifies the event type (earnings beat / debt issuance / enforcement);
extracts material figures & terms (EPS + surprise %; note size, coupon, maturity; "no guidance");
applies domain knowledge — debt issuance → refinancing / capital structure; the NASD qualified-underwriter rule; "no operational catalysts → neutral";
forms a calibrated judgment (direction + materiality) derived from that reasoning.

Crucially, the reasoning field comes before direction / score_bucket in the JSON — a short "think-then-label" chain, so the label is the output of the reasoning, not a standalone guess.

But — honestly — this is reading-depth, not analyst-depth. By design it does NOT:

synthesize across multiple documents (one text, single pass);
do valuation, scenario, or deep multi-step chain-of-thought (no long <think> like math);
run time-series / cross-sectional analysis.

Why concise? (1) The teacher target is deliberately 2–4 factual, ex-ante sentences (no deliberation / <think>). (2) The base model's deep <think> math reasoning was traded away during SFT — that is the measured forgetting (GSM8K 94→67). We built a reader, not an analyst — on purpose.

This depth is exactly right for a bulk reader (raw text → structured record; 87–88% teacher agreement; 100% valid JSON). And deeper reasoning would not help prediction anyway — the 35B teacher and the 12B model are also at chance (the ceiling is the signal, not the reasoning). For deeper financial reasoning you would train a different recipe (keep the <think> CoT, a multi-step / multi-document target, or reasoning-distillation) — a possible next generation.

6. Evaluation — capability & the honest negative result

General capability (apples-to-apples, 150 GSM8K / 80 MBPP / robustness 40×5):

	Teacher 35B	Student 3B	Gemma-12B	Base VibeThinker-3B
GSM8K (math)	94.0%	67.3%	96.7%	94.0%
MBPP (code)	50%	41.2%	84%	12%
robustness (dir consistency)	98%	94%	99%	—

The student traded some general math (94→67, full-FT erosion) for finance specialization; code intact.

🔴 Predictive test (the real finance question): NO EDGE — and that's the finding. On a 147-record walk-forward holdout with realized T+20 returns (chance = 50%):

	Teacher 35B	Student 3B	Gemma-12B
sign-accuracy	39.6%	36.6%	39.6%
IC	+0.192	+0.171	+0.160
P&L	−1.02	−1.53	−1.10

All three — a 35B, a 12B, and this 3B — are at chance. The ceiling is the label/signal (sentiment ≠ alpha), not model size or intelligence. Confidence does not predict returns either (flat calibration). Use this model for structured reading/extraction, not as a trading signal. A real predictive edge would need RLVR grounded on realized returns (future work), not more SFT.

Training status — SFT complete; RLVR (GRPO) attempted but NOT feasible on this hardware

This model is the SFT deliverable (a finance reader). The intended next stage — RLVR / GRPO grounded on the realized-return sidecar (reward = does the ex-ante call match the realized T+20 excess move?) — was attempted but could not be completed on the training box (WSL · 2× RTX 3090 / 48 GB):

recurring WSL CUDA "device not ready" / out-of-memory failures — online RL with K-rollouts per prompt is far heavier than SFT or inference, and 48 GB was not enough;
the runs that did start did not learn: most GRPO groups had zero reward variance (frac_reward_zero_std ≈ 70%), i.e. the realized-return reward was too faint to produce a gradient — consistent with the label-ceiling result above.

So RLVR is future work for a bigger box (A100 / native Linux) with a stronger reward design. The reward function, the realized-return data, and the GRPO scripts are all prepared — only the compute is the blocker. The shipped SFT reader is complete and fully evaluated.

7. Quantization — every type tested (same eval suite)

All variants were run through the identical student suite. "Finance-lossless" = valid-JSON 100% and direction/score within bf16 noise.

✅ Working (shipped here)

variant	format	bits	size	valid-JSON	dir-3way	GSM8K	MBPP
bf16 (reference)	—	16	6.2 GB	100%	88%	67%	41%
W4A16	compressed-tensors (AWQ)	4	2.05 GB	100%	87%	63%	39%
W8A16	compressed-tensors (GPTQ)	8	3.44 GB	100%	87%	69%	43%
GGUF Q8_0	GGUF	8	3.29 GB	100%	88%	69%	45%
GGUF Q6_K	GGUF	6	2.54 GB	100%	86%	72%	43%
GGUF Q5_K_M	GGUF	5	2.22 GB	100%	87%	64%	48%
GGUF Q4_K_M	GGUF	4	1.93 GB	100%	86%	69%	45%
GGUF Q3_K_M (imatrix)	GGUF + imatrix	3	1.59 GB	99%	87%	55%	43%
GGUF Q3_K_L (imatrix)	GGUF + imatrix	3	1.71 GB	99%	87%	58%	38%
GGUF Q2_K (imatrix)	GGUF + imatrix	2	1.27 GB	100%	87%	51%	10% ⚠️

⚠️ Q2_K (2-bit) is finance-lossless (valid-JSON 100%, dir-3way 87%, sign-acc 40%) but general code collapses (MBPP 41→10%) and math drops (67→51%) — ship it for the finance-reading task only, not as a general model. It is the smallest usable variant at just 1.27 GB.

Also lossless but not shipped as a file (runtime-only): bitsandbytes NF4 (vLLM --quantization bitsandbytes).

❌ Did NOT work on this hardware (documented, not shipped)

variant	why
FP8 (W8A8 + weight-only)	RTX 3090 / Ampere has no FP8 tensor cores (FP8 needs Hopper/Ada) → garbage output
GPTQ-4bit	broke this small model (0 valid) while AWQ-4bit is lossless → for a 3B, AWQ-4bit ≫ GPTQ-4bit
GGUF Q3/Q2 without imatrix	sub-4-bit destroys a 3B → an importance matrix (imatrix) rescues 3-bit (see above)

Key quant findings:

4-bit is enough for the finance task — AWQ-W4A16, GGUF-Q4_K_M, bnb-NF4 are all lossless.
Low-bit GGUF (≤3-bit) only works with an imatrix (importance matrix from in-domain finance calibration). Without it, Q3/Q2 produce pure garbage (0% valid). With an imatrix, even 2-bit (1.27 GB) is finance-lossless — though at 2-bit general code ability collapses, so Q2 is finance-reading-only.
FP8 is a Hopper/Ada format — don't use it on Ampere.
Quantization does not speed up this small model at batch (Q4-Q8 TP1 ≈ 900–1175 tok/s vs bf16 1175) — the win is VRAM/density, not per-stream speed. For max throughput on 2 GPUs use data-parallel TP=1 copies, not TP=2 (TP=2 only ~1.2–1.5× for a 3B). See QUANT_COMPARISON.md.

8. Usage

vLLM (compressed-tensors):

vllm serve <repo>/w4a16          # or w8a16 — quantization auto-detected

vLLM (GGUF):

vllm serve <repo>/gguf/student-Q4_K_M.gguf --tokenizer <base-or-this-repo>

llama.cpp (GGUF):

llama-cli -m student-Q4_K_M.gguf -p "<chat-formatted prompt>"

Prompt the model exactly as it was trained: "You are a financial analyst. Read ONLY the text below and return STRICT JSON … (ex-ante, no future prices)" — then the text. Use the model's chat template. Serve at 8192 context for long filings (its only weak spot is truncation on very long sec_fulltext).

9. Limitations

Not a return/price predictor — at chance on realized returns (so is the 35B teacher and Gemma-12B). Use for reading.
Long-document truncation — sec_fulltext agreement drops to ~69%; serve at 8192 ctx to mitigate.
Survivorship / short news window in the underlying labels (current S&P 500 only; news ~3–4 months) — affects any predictive use, not the reading task.
Quality ceiling = the teacher's labels (no human gold). FP8 / GPTQ-4bit / non-imatrix ≤Q3 quants are broken (above).

10. Files in this repo

bf16/ — full-precision SFT source weights (6.2 GB, 2 safetensors shards) — the reference model every quant is derived from; load directly with transformers/vLLM.
w4a16/, w8a16/ — compressed-tensors (vLLM, auto-detected).
gguf/student-Q4_K_M · Q5_K_M · Q6_K · Q8_0 — standard K-quants (lossless).
gguf/student-Q3_K_M-imat · Q3_K_L-imat — imatrix 3-bit (finance-lossless).
gguf/student-Q2_K-imat — imatrix 2-bit, 1.27 GB (finance-lossless; smallest — but general code degraded, finance-only).

Base: VibeThinker-3B (license inherits from the base model). Teacher: Qwen3.6-35B-A3B. Finance-SFT, distillation, and quantization: 2026-06. Quantized with in-domain finance calibration (llmcompressor / llama.cpp imatrix).

Training data: BatuhanECB/vibethinker-3b-finance-sftmini-data · Code & reports: github.com/batuhan3526/vibe_train

Downloads last month: 45

GGUF

Model size

3B params

Architecture

qwen2

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit