PaddySeahorse/tinyllama-4bit

TinyLlama-1.1B-4bit

This repo contains a 4-bit quantized version of the TinyLlama-1.1B base model, quantized in the summer of 2025. It is designed for ultra-lightweight inference, edge computing, and hardware benchmarking.

Model Description

• Base Model: TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T (or equivalent 1.1B base)
• Quantization Precision: 4-bit
• Format: PyTorch / bitsandbytes compatible
• Language Support: Primarily English (with limited multilingual embedding capabilities)

Intended Uses & Limitations

Best For:

• Edge Computing: Ideal for low-resource devices like Raspberry Pi, older smartphones, or embedded systems.
• Prototyping & Benchmarking: Perfect for testing inference pipeline setups (transformers, accelerate) without high VRAM/RAM overhead.
• CI/CD Testing: A lightweight sandbox model for automated testing scripts.

Limitations:

• Due to the aggressive 4-bit quantization on a 1.1B parameter base, the model may exhibit degradation in complex reasoning or long-text generation.

Quick Start (Usage)

You can easily load and run this model using Hugging Face transformers and bitsandbytes:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "PaddySeahorse/tinyllama-4bit"

# 4-bit configuration
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_quant_type="nf4"
)

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    quantization_config=bnb_config, 
    device_map="auto"
)

# Inference
prompt = "The meaning of life is"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=40)
    print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Created in Summer 2025 as an exploration in LLM quantization.