SlayerLab/lojban-gpt2-rsi1

Podziękowania

Moc obliczeniowa dzięki uprzejmości Comtegra GPU Cloud Strona główna | Comtegra GPU Cloud Chmura GPU | Zasoby obliczeniowe GPU | GPU na żądanie. Comtegra GPU Cloud to sposób na interakcję z zasobami GPU. Korzystaj z mocy klastra bez konieczności znajomości Kubernetesa

Comtegra GPU Cloud

Lojban GPT-2 RSI-1

A 97.7M parameter GPT-2 model trained on the Lojban constructed language using Recursive Self-Improvement (RSI).

Model Details

Parameter	Value
Architecture	GPT-2 (decoder-only transformer)
Parameters	97.7M
Layers	12
Hidden dim	768
Attention heads	12
Vocab size	8,000 (custom BPE)
Context length	512 tokens
Training iterations	50,000
Training tokens	181M
Real data ratio	33%
Best validation loss	2.5805
camxes grammaticality	76.6%

Training Methodology: Recursive Self-Improvement (RSI)

This model was trained using a Recursive Self-Improvement loop:

1. V8 (base model) generates 50,000 Lojban sentences
2. Sentences are filtered through the camxes PEG parser (23.7% pass rate)
3. 11,260 grammatical sentences are combined with:
   • 438K real Lojban sentences (10× upsampled)
   • 8.9M synthetic sentences from V8 corpus
4. Total: 181M tokens, 33% real data ratio
5. Train new model (RSI-1) for 50K iterations

Evaluation

Validation Loss Comparison

Model	Val Loss	camxes %
V3 (5M synthetic)	~4.5	0.3%
V4 (Metaspace tokenizer)	3.80	26.7%
V5 (GPT-2 small)	3.03	87.4%
V8 (99.3% synthetic)	3.66	—
RSI-1 (this model)	2.58	76.6%

camxes Grammaticality

500 generated sentences evaluated with the camxes PEG parser:

• 383/500 grammatical (76.6%)
• Average words/sentence: 16.2

Sample Generations

Grammatical (PASS):

✓ coi rodo mi'e la bripre
✓ dei notci fo do mu'i le nu do satci casnu dei kei fe levi tarci poi lei cevni coi
✓ tu cu nimre lu ckule sisti li'u ko di'a vofli le'e ganse doi cnt
✓ lo prenu cu tavla bau lo ponjo
✓ la .alis. cu viska lo cinfo

Usage

import torch
from tokenizers import Tokenizer

# Load tokenizer
tok = Tokenizer.from_file("tokenizer.json")

# Load model
checkpoint = torch.load("model.pt", map_location="cpu")
model.load_state_dict(checkpoint)

# Generate
input_ids = torch.tensor([[tok.encode("<|bos|>mi klama lo zdani").ids]])
# ... standard GPT-2 autoregressive generation

Training Data

Trained on the SlayerLab/lojban-master dataset — the largest publicly available Lojban corpus (542K sentences, 7.9M words from 15 sources).

Limitations

• RSI-1's camxes score (76.6%) is lower than V5 (87.4%) despite better val loss (2.58 vs 3.03). This is because:

  • RSI-1 generates longer, more complex sentences (avg 16.2 words vs V5's shorter outputs)
  • More complex sentences have more opportunities for grammatical errors
  • V5's simpler outputs were easier to keep grammatical

• The model was trained on a single RTX 3090 (24GB) over ~22 hours.

License

MIT