💨 Vapor-2B-V1.2

Vapor-2B-V1.2 is a specialized, lightweight causal language model built with a high-efficiency Hybrid Short-Conv & Full-Attention architecture. Engineered specifically for high-throughput, low-latency, and memory-constrained inference, Vapor-2B-V1.2 delivers fast token generation with an exceptionally compact memory footprint.

⚡ Key Architectural Specifications

• Model Name: Vapor-2B-V1.2
• Model Size: ~2.0 Billion parameters (1.95B linear parameters)
• Architecture: Hybrid Linear Short-Conv and Full-Attention
• Total Layers: 30 layers (22 Causal 1D Depthwise Conv layers + 8 Multi-Head Attention layers)
• Hidden Dimension: 2,048 ($D_{\text{ffn}} = 10,752$ with SwiGLU activations)
• Attention Mechanism: Grouped-Query Attention (32 Q-heads, 8 KV-heads) with Rotary Position Embeddings (RoPE)
• Context Length: Native support up to 131,072 tokens (128k context length)
• Memory Footprint: ~4.3 GB in BF16/FP16 (Runs effortlessly on consumer GPUs like RTX 3050, RTX 4060, and edge devices)

🚀 Quickstart & Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Neeze/Vapor-2B-V1.2"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
 )

prompt = "Explain the architectural benefits of combining 1D convolutions with self-attention in language models:"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=200,
        do_sample=True,
        temperature=0.7,
        top_p=0.9
    )

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

📋 Intended Use

Vapor-2B-V1.2 is optimized for developers needing fast, low-latency text generation, agentic reasoning, and interactive conversation on resource-limited GPU environments without sacrificing context window length.