CNWPlayer/gemmeh-it-GGUF

Have some GGUFs idk

Wow, my first quantizations! Thanks GPT 5.6 Luna for helping me with this (i'm an idiot). These GGUF files were converted from the original Hugging Face checkpoint using the Gemmeh-compatible llama.cpp implementation.

To actually do anything with these files, you need this fork of llama.cpp.

Also have some graphs :D

---

Gemmeh-IT 1B

Gemmeh-IT is the instruction-tuned version of ni-co-la-s/gemmeh, a 1.1B-parameter decoder-only transformer trained for educational purposes from scratch on 20B tokens of pre-2024 FineWeb-Edu, then LoRA-finetuned on OpenHermes for chat.

Architecture is Gemma 3-inspired without sliding-window attention. Custom 32k SentencePiece BPE tokenizer trained on the same corpus.

For the base model, see ni-co-la-s/gemmeh. For the GGUF, see ni-co-la-s/gemmeh-it-GGUF.

Usage

This model uses custom modeling code, so trust_remote_code=True is required.

import torch
from huggingface_hub import hf_hub_download
from sentencepiece import SentencePieceProcessor
from transformers import AutoModelForCausalLM

repo_id = "ni-co-la-s/gemmeh-it"

model = AutoModelForCausalLM.from_pretrained(
    "ni-co-la-s/gemmeh-it",
    trust_remote_code=True,
    torch_dtype="bfloat16",
)
model.eval()
print("Model loaded")

# Load tokenizer
sp_path = hf_hub_download(
    repo_id="ni-co-la-s/gemmeh-it",
    filename="tokenizer.model",
    token=True,
)
sp = SentencePieceProcessor()
sp.Load(sp_path)
print("Tokenizer loaded")

# Test generation
def generate(question, max_new_tokens=40, temperature=0.0):
    prompt = (
        f"<start_of_turn>user\n{question}\n<end_of_turn>\n"
        "<start_of_turn>model\n"
    )
    ids = sp.Encode(prompt, out_type=int)
    input_ids = torch.tensor([ids], dtype=torch.long)
    with torch.no_grad():
        for _ in range(max_new_tokens):
            out = model(input_ids)
            next_logits = out.logits[0, -1, :]
            if temperature == 0:
                next_id = next_logits.argmax().item()
            else:
                probs = torch.softmax(next_logits / temperature, dim=-1)
                next_id = torch.multinomial(probs, 1).item()
            if next_id == sp.eos_id():
                break
            input_ids = torch.cat([input_ids, torch.tensor([[next_id]])], dim=1)
    return sp.Decode(input_ids[0][len(ids):].tolist())

print(generate("What is the capital of France?", max_new_tokens=40, temperature=0.0))

Training details

Parameters	1.1B
Architecture	Gemma 3-inspired
Vocab	32,768 (SentencePiece BPE, English-only)
Context	4,096
Pretraining data	FineWeb-Edu sample, 20B tokens
Knowledge cutoff	Pre-2024 (intentional)
Finetuning	LoRA rank 16 on OpenHermes (250M assistant tokens)

Benchmarks

Evaluated through the BF16 GGUF served via llama.cpp with lm-eval. Gemma 3 1B IT numbers obtained locally by running unsloth/gemma-3-1b-it-GGUF at Q8_0 through the same pipeline.

Benchmark	Metric	Gemmeh 1B (base)	Gemmeh-IT 1B	Gemma 3 1B IT (local)
PIQA	0-shot	70.2	71.4	72.8
ARC-Challenge	25-shot	38.4	40.4	40.3
ARC-Easy	0-shot	57.3	57.6	63.4
WinoGrande	5-shot	52.2	54.0	55.1
TruthfulQA	mc2, 0-shot	37.8	44.8	38.9

Trained on roughly 10-100x less data than the references, with a much smaller (32k vs 262k) vocabulary, and no distillation.

Limitations

• Smaller and less benchmark-competitive than similarly-sized models trained on more data.
• English only.
• 4,096 token context.
• Pre-2024 knowledge only.