Mnemosyne-64M-Instruct (Instruction-Aligned)

Mnemosyne-64M-Instruct is the conversationally aligned instruction model built on the Hierarchical Chunk Attention (HCA) architecture.

It was pre-trained from scratch on 1.28 Billion tokens of educational text and subsequently aligned on 25,000 multi-turn conversations from UltraChat-200k.

Training Datasets

1. Pretraining Corpus: HuggingFaceFW/fineweb-edu
   • Volume: 1.28 Billion Tokens (Educational and synthetic reasoning text).
2. Instruction Alignment (SFT): HuggingFaceH4/ultrachat_200k
   • Volume: 25,000 multi-turn dialogues (~20.5M tokens with masked user prompt loss).
   • SFT Loss: 2.4593 (Outperformed our 64M Transformer baseline of 2.4813 on identical data).

Prompt Format

The model expects standard instruction framing:

<|user|>
{user_prompt}
<|assistant|>

How to Run Inference

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BIBLIOKLEPT/Mnemosyne-64M-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    dtype=torch.bfloat16, 
    device_map="cuda", 
    trust_remote_code=True
)

prompt = "<|user|>\nGive me three practical tips for eating healthy.\n<|assistant|>\n"
inputs = tokenizer(prompt, return_tensors="pt").to("cuda")

with torch.no_grad():
    outputs = model.generate(
        **inputs, 
        max_new_tokens=150, 
        do_sample=True,
        temperature=0.6, 
        top_k=40,
        repetition_penalty=1.15
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))