Cabbache/Fredu-1.7B-Instruct

Fredu 1.7B Instruct

A Maltese-specialised model built from utter-project/EuroLLM-1.7B in two stages on a single consumer GPU (RTX 5060 Ti, 16 GB).

It is fluent in Maltese but hallucinates and does not always reply with anything related to the prompt. It's able to translate english sentences into a related sentence in Maltese. It's quite bad at it but the Maltese translation is related somehow.

How to run

Ollama

ollama run hf.co/Cabbache/Fredu-1.7B-Instruct

The prompt template and sampling parameters are stored in the repo, so nothing else is needed. Pick a build with a tag:

tag	size	notes
(none) / :Q4_K_M	1.0 GB	4-bit, the default.
:Q8_0	1.8 GB	8-bit, close to full quality
:F16	3.3 GB	no quantization

transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("Cabbache/Fredu-1.7B-Instruct")
model = AutoModelForCausalLM.from_pretrained("Cabbache/Fredu-1.7B-Instruct",
                                             dtype="bfloat16", device_map="auto")

q = "X'inhu l-Kunsill Lokali f'Malta?"
ids = tok(f"Mistoqsija: {q}\nTweġiba:", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=200, temperature=0.3, do_sample=True)
print(tok.decode(out[0], skip_special_tokens=True))

There is no chat template. Use the Mistoqsija:/Tweġiba: framing above, which is what the model was trained on. A low temperature helps: what factual knowledge it has is held with very little margin and does not survive sampling at 0.7.

What was done

stage	data	result
Continued pretraining	336M tokens, 125,900 documents from Korpus Malti	perplexity 8.29 → 5.98
Supervised fine-tuning	43,988 Maltese question/answer pairs	answer-loss 1.70 → 1.40

Both stages were full fine-tunes (all 1.657B parameters), bf16 weights and gradients with 8-bit AdamW and gradient checkpointing, ~14 GB VRAM.

Prompt format

The model was trained on this exact framing, with loss computed only on the answer:

Mistoqsija: {question}
Tweġiba: {answer}

It reads informal Maltese and replies in correct Maltese

35% of training questions had their diacritics stripped (ħ→h, ġ→g, ċ→c, ż→z, so għ→gh) while every answer was left correct. Because loss lands only on the answer, the model is shown informal spelling and never rewarded for producing it.

Q: X'inhu n-numru medju ta' sighat ta' rqad li persuna ghandha tiehu?
A: In-numru medju ta' sigħat ta' rqad li persuna għandha tieħu huwa madwar
   7.5 sigħat kuljum.

Note sighat → sigħat, ghandha → għandha, tiehu → tieħu.

Evaluation

Perplexity on held-out documents, by register, against the untrained base:

source	base	this model	change
government gazette	7.84	2.97	−62.1%
parliament	6.89	3.19	−53.6%
press	7.72	6.01	−22.1%
academic	12.61	10.38	−17.7%
blogs	10.43	8.93	−14.4%
wikipedia	6.51	6.06	−6.9%

A 57-item Malta factual eval (greedy decoding, an upper bound on knowledge):

	this model	base
overall	27/57	22/57
history	12/13	8/13
geography	4/17	6/17

Licence

Do whatever you want with it