LFM3-1.7B-Turkish (LoRA continued-pretraining)

A Turkish-tuned fine-tune of LiquidAI/LFM3-1.7B via LoRA continued-pretraining on Turkish prose. LFM3-1.7B is a hybrid liquid-LLM (conv + sliding/full attention) with an LFM Open License; it is not supported by MLX, so this fine-tune was run with PyTorch + TRL SFTTrainer + PEFT LoRA.

Summary

This model improves Turkish language-modelling over the base LFM3-1.7B. On a held-out eval split from the same corpus, held-out next-token perplexity dropped from 26.17 to 8.28 (loss 3.264 → 2.113).

	Held-out Turkish PPL
Base LiquidAI/LFM3-1.7B	26.17
After LoRA CPT	8.28

Data

• Corpus: tascib/turkish-llm-dataset (Turkish, CC-BY-SA-4.0), 10,000 train examples streamed from the train split.
• Task: causal language modeling (continued pretraining) on raw text.
• Held-out evaluation: 30 examples, next-token loss / perplexity.

How it was trained

• Base model: LiquidAI/LFM3-1.7B
• LoRA: PEFT rank r=16, alpha=32, dropout=0.05, task_type=CAUSAL_LM, targets: out_proj, w2, v_proj, up_proj, w1, o_proj, q_proj, in_proj, w3, k_proj, down_proj, gate_proj.
• Recipe: TRL SFTTrainer on the text field, bf16 model, fp32 LoRA, cosine LR, run on Apple Silicon (CPU) — the model is not MLX-compatible.
• Outputs: LoRA adapter (lora/) and a merged, standalone model (this repo root).

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
tok = AutoTokenizer.from_pretrained("bedop/LFM3-1.7B-Turkish", trust_remote_code=True)
# ...or apply the LoRA to the base:
from peft import PeftModel
base = AutoModelForCausalLM.from_pretrained("LiquidAI/LFM3-1.7B", trust_remote_code=True, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(base, "bedop/LFM3-1.7B-Turkish/lora")

License

LFM Open License v1.0 (inheriting the base model). See LICENSE. Dataset: CC-BY-SA-4.0.

Created by Hermes Agent · fine-tune pipeline in scripts/.