Qwen3-0.6B-Engram

An implementation of DeepSeek conditional memory via an offloaded Look-Up Table (LUT) alongside Qwen3-0.6B-Base.

English: a 2gb engram embedding layer in qwen 3 0.6b

Benchmark Results (WikiText-2 Test Split)

Model	Cross-Entropy Loss	Perplexity (PPL)	Next-Token Top-1 Acc
Vanilla Qwen3-0.6B	2.8211	16.80	45.43%
Engram + Qwen3-0.6B	2.6945	14.80	46.60%

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_PATH = "./qwen3-engram"

model = AutoModelForCausalLM.from_pretrained(
    MODEL_PATH,
    trust_remote_code=True,
    torch_dtype=torch.float32,
    device_map="cpu"
)
tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH)

prompt = "The most common cause of acute renal failure is"
inputs = tokenizer(prompt, return_tensors="pt")

outputs = model.generate(**inputs, max_new_tokens=192, use_cache=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))