🧟 Frankenstein-3.0: The Unified AI Entity

A production-grade multi-modal AI orchestration system with a trained routing brain, featuring 4 specialist models, 5 integrated tools, and triple-layer fault tolerance.

Modality: Text → Text • Image • Video • Audio (orchestrated across specialists) Repo weights: RouterBrain (text router, QLoRA-fine-tuned Qwen2.5-1.5B)

Architecture Overview

┌─────────────────────────────────────────────────────────────┐
│                     USER REQUEST                            │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  LAYER 1: Pre-Flight Regex (100% accurate, hard patterns)   │
│  • "uploaded file" → RAG    • "then" → multi-step chain     │
│  • "explain how to" → chat                                  │
└─────────────────────────────────────────────────────────────┘
                            ↓ (pass-through)
┌─────────────────────────────────────────────────────────────┐
│  LAYER 2: RouterBrain (0.3s instant routing)                │
│  • Qwen2.5-1.5B fine-tuned with QLoRA, merged to fp16       │
│  • Outputs JSON routing decisions                           │
└─────────────────────────────────────────────────────────────┘
                            ↓ (if brain fails)
┌─────────────────────────────────────────────────────────────┐
│  LAYER 3: Qwen3.5-4B Resident (smart fallback)              │
│  • Understands complex/novel requests                       │
└─────────────────────────────────────────────────────────────┘
                            ↓
┌─────────────────────────────────────────────────────────────┐
│  SPECIALISTS:  🧠 Chat   💻 Code   🎨 Image   🎬 Video       │
└─────────────────────────────────────────────────────────────┘

The RouterBrain (This Repo's Weights)

The root weights are the RouterBrain: a distilled routing model that reads user requests and outputs JSON routing decisions in ~0.3 seconds.

Load the RouterBrain

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Questionmarkboy/frankenstein-3.0")
model = AutoModelForCausalLM.from_pretrained(
    "Questionmarkboy/frankenstein-3.0",
    torch_dtype="float16",
    device_map="auto"
)

prompt = "Route to the correct specialist (chat/code/image/video/search/calc/execute/rag). USER: Draw a robot then animate it dancing. Respond ONLY with JSON."

text = tokenizer.apply_chat_template(
    [{"role": "user", "content": prompt}],
    tokenize=False,
    add_generation_prompt=True
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=120, do_sample=False, repetition_penalty=1.05)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Expected Output

{"specialist": "multi", "confidence": 0.95, "reasoning": "Chained multi-step task", "steps": [{"specialist": "image", "prompt": "Draw a robot"}, {"specialist": "video", "prompt": "animate it dancing"}]}

The Full System (app.py)

Specialists

Specialist	Model	Use Case
Chat	Qwen3.5-4B	Conversation, reasoning, explanations
Code	Qwen2.5-Coder-14B	Programming, debugging, code generation
Image	SDXL	Photorealistic image generation
Video	MiniMax-H3 NF4	Video + audio generation

Tools

🔍 Web Search — DuckDuckGo real-time search
🧮 Calculator — SymPy safe mathematical evaluation
⚡ Code Execution — Sandboxed Python execution (5s timeout)
📚 RAG — Document upload + keyword search
🧠 Memory — SQLite persistent conversation history

Installation

Prerequisites

Requirement	Minimum	Notes
Python	3.10+	3.11 recommended
pip	latest	upgraded in Step 3
Git + Git-LFS	any	LFS needed for the ~3GB weights
RAM	16 GB	32 GB recommended
GPU (optional)	8 GB VRAM	16-24 GB VRAM for image/video specialists
CUDA (optional)	12.1+	only required for GPU features

Step 1 — Clone the repository

git lfs install
git clone https://huggingface.co/Questionmarkboy/frankenstein-3.0
cd frankenstein-3.0

Step 2 — Create a virtual environment (recommended)

python -m venv venv
source venv/bin/activate        # Linux / macOS
venv\Scripts\activate          # Windows

Step 3 — Install dependencies

pip install --upgrade pip
pip install -r requirements.txt

Step 4 — Launch the app

python app.py

Then open http://localhost:7860 in your browser.

CPU-only machines

Installation is identical. The app auto-detects the absence of CUDA and runs the RouterBrain, chat and code specialists on CPU. Image and video specialists respond with a friendly "GPU required" message instead of crashing.

Troubleshooting

bitsandbytes warnings on CPU — expected and safely ignored by the app.
Slow clone — the RouterBrain weights are ~3 GB; make sure Git-LFS is installed (Step 1).
Port 7860 already in use — change server_port at the bottom of app.py.
Out of memory on GPU — the app loads specialists on demand and unloads them after use; close other GPU processes or reduce image resolution in app.py.

Hardware Requirements

Mode	Minimum	Recommended
RouterBrain only	2GB VRAM	CPU works fine
Chat + Code (CPU)	16GB RAM	32GB RAM
Full GPU stack	16GB VRAM	24GB+ VRAM

The app auto-detects CUDA. On CPU-only machines, chat/code work and image/video return a friendly "GPU required" message.

Training Details

RouterBrain Training

Base Model: Qwen2.5-1.5B-Instruct
Fine-tuning: QLoRA (r=32, α=64, dropout=0.05)
Dataset: ~10k synthetic routing examples (combinatorial diversity, noise injection, 30+ edge-case patterns)
Training: 2-3 epochs on T4 GPU
Merge: Clean fp16 merge (no quantization rounding)

Performance Metrics

Pre-flight layer: 100% accuracy on hard patterns
RouterBrain: 60% standalone accuracy on diverse test set
Effective routing: 90%+ via triple-layer fault tolerance

Multi-Step Task Chaining

User: "Draw a robot then animate it dancing"
  ↓
Pre-flight: detects "then" → routes to multi specialist
  ↓
Execution: image specialist → video specialist (sequential)
  ↓
Output: generated image + animated video with audio

Architecture Decisions

Why Triple-Layer Routing?

Speed: Pre-flight regex is instant (0.001s)
Efficiency: RouterBrain is fast (0.3s) vs prompt-based (4-6s)
Robustness: 4B fallback catches edge cases
Production-ready: Graceful degradation under all conditions

Why Distill A Routing Brain?

20x faster than prompt-based routing
50% less VRAM than using the 4B resident for routing
Perfect JSON output (trained to only output JSON)
Cheaper to deploy at scale

Companion Models

Frankenstein-2.0 — Merged Mistral-7B checkpoint

Technical Stack

Framework: HuggingFace Transformers + Diffusers
Quantization: BitsAndBytes (NF4, 4-bit)
UI: Gradio 4.44
Memory: SQLite
Training: PEFT + LoRA

Known Limitations

Image/video generation requires 16GB+ VRAM
RAG only supports .txt files (no PDF/DOCX yet)
Web search is basic (DuckDuckGo HTML scraping)
Code execution has 5s timeout for safety

Citation

@misc{frankenstein3,
  author = {Questionmarkboy},
  title = {Frankenstein-3.0: Multi-Modal AI Orchestration System},
  year = {2024},
  publisher = {HuggingFace},
  journal = {HuggingFace Repository},
  howpublished = {\url{https://huggingface.co/Questionmarkboy/frankenstein-3.0}}
}