Ayachi7/CallVerse-Qwen3VL-2B-Vision-QLoRA

CallVerse Vision — Qwen3-VL-2B QLoRA Adapter

A domain-adapted, parameter-efficient vision-language adapter for visual evidence in e-commerce and tourism customer-support workflows.

This repository contains only the LoRA adapter (not the full 2B-parameter base model). It was trained in 4-bit NF4 (QLoRA) from Qwen/Qwen3-VL-2B-Instruct. It extracts visible facts from a photograph, returning compact JSON for a downstream support system; it does not determine a customer's intentions, emotions, booking status, or refund eligibility.

Research prototype — not production-validated. Results below come from a 25-image held-out test split of a small curated dataset. They do not establish reliability for arbitrary customer photos. The adapter does not authorize customer-impacting actions.

At a glance

Field	Value
Base model	Qwen/Qwen3-VL-2B-Instruct
Base revision	89644892e4d85e24eaac8bacfd4f463576704203
Adaptation	QLoRA (4-bit NF4 training), PEFT adapter
LoRA rank / alpha / dropout	16 / 32 / 0.05
Fine-tuned parameters	Approximately 17.4M; language model adapters, not the vision encoder
Tasks	Visual product damage, baggage damage, hotel-room condition, visible ambiguity
Training data	103 images; 22 validation; 25 test (150 total)
Result	84% exact issue-type accuracy, 79.15% macro-F1 on held-out test
Comparison	64% exact accuracy with improved prompting of the unmodified base model
Training	Kaggle, 2× Tesla T4 available; FP16 compute, NF4; selected rank-16 adapter
Application	CallVerse: ShopNova (e-commerce/retail), Atlas Travel (tourism)

Evaluation

All results below were read from the Kaggle training artifact benchmark_comparison.csv and are not independently re-run for this release.

Model configuration	Held-out n	Exact issue-type accuracy	Macro-F1	Damage recall	JSON validity	Mean latency
Base, improved structured prompt	25	64%	61.37%	100%	100%	3.796 s
QLoRA adapter, same task	25	84%	79.15%	100%	100%	6.405 s

Accuracy gain is +20 percentage points (21/25 vs. 16/25). The notebook's bootstrap interval for the paired gain is +8 to +36 percentage points under its resampling protocol. With 25 test images, one error changes accuracy by four points; these statistics must not be interpreted as proof of deployment robustness.

Damage recall is for the notebook's binary damage classification, not eight-class recall or perfect recognition of all damaged items. Some classes have 1–4 test examples. The original unstructured prompt scored 0% on strict JSON schema validation, but that does not mean it recognized no damage; it produced nonconforming issue names. Comparisons against that zero are misleading.

The dataset was curated from publicly reachable sources; 150 photos were marked real, 0 synthetic by the preparation report. Some image origins and redistributability remain restricted or uncertain, so the underlying photographs are not published in this model repository. Additionally, customer-context messages used for future routing tests were AI-authored and were not used to train the vision adapter. These limitations apply to any evaluation based on this collection.

Labels

• cracked_screen
• screen_display_defect
• damaged_packaging
• damaged_luggage
• dirty_or_messy_room
• damaged_room_fixture
• no_visible_issue
• unclear

The adapter produces a structured assessment with keys visible_damage, object, issue_type, description, confidence_level, and requires_review.

Quick start

Install current compatible versions of transformers, peft, accelerate, torch, Pillow and optionally bitsandbytes. Run pip install -r requirements-inference.txt from this repository.

from PIL import Image
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
from peft import PeftModel
import torch

BASE = "Qwen/Qwen3-VL-2B-Instruct"
ADAPTER = "YOUR_HF_USERNAME/CallVerse-Qwen3VL-2B-Vision-QLoRA"  # replace with the public repository ID
processor = AutoProcessor.from_pretrained(BASE)
base = Qwen3VLForConditionalGeneration.from_pretrained(
    BASE, torch_dtype=torch.float16, device_map="auto"
)
model = PeftModel.from_pretrained(base, ADAPTER)
model.eval()

# See inference.py for the exact training-matched task prompt, preprocessing and JSON handling.

For full executable inference, run python inference.py --repo-id YOUR_HF_USERNAME/CallVerse-Qwen3VL-2B-Vision-QLoRA --image path/to/photo.jpg. The full base model must be downloadable and sufficient VRAM/RAM must be available. Inference was not re-executed in this publication package.

How it fits CallVerse

Image ──────────────> Qwen3-VL adapter ─────> structured visual evidence ──┐
Customer message ──> Intake (intent, sentiment, emotion) ───────────────────┤
                                                                          ↓
                                                                    Late fusion
                                                                          ↓
                                                           Queue Router → Agents
                                                                          ↓
                                                         Coordinator + policy RAG

Visual evidence is supplementary and unverified: the Coordinator and business systems must verify account facts and authorized actions. A photo alone does not imply an intention to cancel, rebook, return or request a refund. Emotion and sentiment should be unknown for image-only requests unless independent customer text supports them.

Training setup

• Objective: image-conditioned supervised instruction fine-tuning, answer-only loss.
• Source: 150 curated photographs across e-commerce and hotel/luggage scenarios.
• Split: 103 train, 22 validation, 25 held-out test; seed 42; related image groups kept in one split.
• QLoRA: NF4 4-bit quantization of base weights, FP16 computation, gradient checkpointing.
• Adapter: LoRA rank 16, alpha 32, dropout 0.05, target projections q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj inside the language model.
• Learning rate: 1e-4; 3 epochs; microbatch 1, accumulation 8.
• Selected on validation macro-F1 with validation loss as tiebreaker.
• Kaggle stack: Python 3.13.15, Torch 2.11.0+cu128, Transformers 5.22.1, PEFT 0.21.2, bitsandbytes 0.50.2.
• Run duration reported by notebook: 74.6 minutes including data and model benchmarking (not 74.6 minutes of just optimization).

Reproducibility artifacts are in evaluation/: configuration, dataset counts, complete benchmark CSV, summary metrics, and confusion matrix. Raw customer photos and the training dataset are deliberately excluded.

Risks and limitations

1. Small test sample (25 images) and sparse classes; highly uncertain per-class metrics.
2. Domain/source confounding: dirty-room photos came from one mobile-home source while clean-room photos came from a different hotel-photo dataset; the adapter may learn source style instead of cleanliness.
3. License/provenance limitations: some training images came from research-only or unclear-license sources. The photos are not redistributed. Review source conditions before commercial deployment or further distribution.
4. Vision hallucinations: a model can misclassify cracks, damage, dirt, or ambiguous photos. Use human review for important decisions.
5. No real-customer production validation, independent large-scale holdout, confidence calibration, or multilingual testing.
6. No authorization: the model cannot determine refund eligibility or confirm customer identity, booking details, or financial records.
7. Latency tradeoff: measured fine-tuned inference was slower than improved-prompt baseline in the notebook environment (6.405 s vs. 3.796 s average).

Responsible use

Suitable for academic prototyping, assisting human review, and generating advisory visual evidence for customer support. Not suitable as a standalone decision maker for refunds, fraud, compensation, accessibility assistance, or other consequential outcomes. Apply business rules, evidence verification and human escalation downstream.

Credits and attribution

CallVerse — advanced Data Science engineering project. Adapter training and evaluation results supplied from the CallVerse Kaggle run. Base architecture and weights are from the Qwen team, linked above. The author and publisher of this adapter should add their Hugging Face name and affiliation in the repository profile; no authorship by upstream dataset creators is implied.

Licensing

The Qwen3-VL base model is available under its own Apache-2.0 terms. This release does not assert that all training image datasets permit commercial use or redistribution. The adapter's redistribution license should be set by its rights holder after reviewing the contributing data terms; the absence of a model-card license tag is intentional until that review is complete. Public access does not automatically grant a commercial-use license.