Article
First wiew from system operating
SoulInPsyAbstract
• Aelin AquaSoul PRO
SoulInPsyAbstract
AI & ML interests
Имею личную экосистему с мулти Гент работами 8 разных млл на разные уровни системы
Recent Activity
updated a model 4 minutes ago
SoulInPsyAbstract/syntax-ai-community published a model 4 minutes ago
SoulInPsyAbstract/syntax-ai-community repliedto their post 5 minutes ago
Fine-Tuning the "Don't Fabricate" Rule: 14 Experiments, One Genuine Signal Over five days we ran 14 fine-tuning experiments across six base models to answer a narrow research question: can a simple behavioural rule — "if you don't know, don't guess or fabricate" — be baked into model weights via fine-tuning, rather than living only in the system prompt?
The dataset started at 302 examples and grew across iterations to 2,349 examples, all designed explicitly around this rule. We tested six models: gpt-4o-2024-08-06, Mistral-7B, Qwen2.5-7B, Phi-3.5-mini, Llama-3.1-8B, and DeepSeek-R1.
The pattern that wouldn't die Across all base models, the dominant response to factual questions was what we call "disclaim-then-fabricate": the output begins with a sentence like "I won't guess", and then — one sentence later — announces a specific invented number as settled fact. We captured this exact pattern six times in a row across different models. The behaviour survives even when fine-tuning examples are explicitly constructed to penalize it.
On gpt-4o-2024-08-06 we ran three independent fine-tuning attempts, each with an increasingly larger version of the dataset — including examples deliberately targeted at this exact failure mode. All three runs continued to fabricate when tested. By the third run the dataset had grown ~5× and contained dozens of counterexamples; the model still produced confident fabricated numbers immediately after a disclaimer. Three out of three attempts, no measurable improvement.
One exception that deserved a hard look Only on the final run — 2,349 examples, deployed via Azure OpenAI (deployment suffix protocol0-v5) — did a single test sample come back completely clean for the first time in the entire series.
Organizations
published an article 4 days ago
published an article 4 days ago
Article
SIPA OS fine tuning experiment
SoulInPsyAbstract
•