FineTunedLlama3 — Fact-Checking & Misinformation Detection (4-bit MLX)

FineTunedLlama3 is an instruction-tuned, 4-bit quantized fact-checking verification model based on Meta-Llama-3.1-8B-Instruct. It is fine-tuned to classify statements and claims as True or False while factoring in contextual metadata such as the speaker and the setting in which the statement was made.

Standalone 4-Bit Fused Model: This repository contains the complete, standalone fused 4-bit model weights (model.safetensors, ~4.3 GB). The LoRA adapters were merged directly into the base model parameters using mlx_lm.fuse. No separate adapter loading is required.

---

Model Details

• Base Model: mlx-community/Meta-Llama-3.1-8B-Instruct-4bit
• Architecture: LLaMA 3.1 (8 Billion Parameters)
• Quantization: 4-bit Group-Quantized (Group size 64) via Apple Silicon MLX
• Tuning Method: QLoRA (Quantized Low-Rank Adaptation) + Full Model Weight Fusion
• Target Task: Binary Fact Verification (True / False) with Speaker & Context Conditioning
• Format: Native MLX / Safetensors weights

---

Dataset & Curation

The model was fine-tuned on the canonical LIAR Benchmark Dataset (Wang, 2017: "Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection), sourced from PolitiFact.

1. Dataset Splits & Statistics

• Training Set: 10,240 verified claims
• Validation Set: 1,284 verified claims
• Test Set: 1,267 verified claims
• Total Samples: 12,791 statements

2. Binary Label Mapping

The original 6-way fine-grained labels were mapped into a rigorous binary ground truth:

Original LIAR Label	Binary Class	Justification
true	True	Factual and accurate statement
mostly-true	True	Accurate with minor clarification
half-true	True	Partially accurate with omitted context
barely-true	False	Substantially inaccurate
false	False	Factually untrue statement
pants-fire	False	Blatantly fabricated / ridiculous lie

3. Prompt Template (ChatML)

Each training sample was structured with system instructions and contextual metadata:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are an expert fact-checking verification system. Your task is to determine whether a given claim is True or False. Analyze the claim in the context of the speaker and the setting in which it was made. Respond with exactly one word: True or False.<|eot_id|><|start_header_id|>user<|end_header_id|>

Claim: {statement}
Speaker: {speaker}
Context: {context}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{True / False}<|eot_id|>

---

Fine-Tuning Setup & Hyperparameters

The model was trained locally on Apple Silicon (16 GB Unified Memory) using the Apple MLX framework (mlx_lm).

Hyperparameter	Value	Description
Fine-Tuning Method	QLoRA	Low-Rank Adaptation on 4-bit base weights
LoRA Rank ($r$)	8	Low-rank matrix dimension
LoRA Alpha ($\alpha$)	16	Scaling parameter ($\alpha/r = 2.0$, scale = 20.0)
Target Layers	16	Top 16 transformer layers (Layers 16–31)
Optimizer	AdamW	Decoupled Weight Decay Regularization
Learning Rate	1e-5	Constant learning rate schedule
Batch Size	1	Micro-batch size optimized for unified memory
Max Sequence Length	512	Token sequence budget
Training Steps	1,000 iterations	Selected at optimal validation convergence
Prompt Loss Masking	True	Loss calculated only on output tokens (True/False)
Gradient Checkpointing	True	Memory-efficient activation recomputation
Seed	42	Deterministic reproducibility
Checkpoint Interval	Every 200 steps	Intermediate adapter safetensors saved

Post-Training Model Fusion

After completing LoRA fine-tuning, the adapter parameters were fused directly into the 4-bit base model:

mlx_lm.fuse \
  --model mlx-community/Meta-Llama-3.1-8B-Instruct-4bit \
  --adapter-path ./adapters \
  --save-path ./fused_model

This produces a unified weight matrix, eliminating adapter routing overhead during inference.

---

Benchmark Evaluation & Results

The fine-tuned model was evaluated on a comprehensive, unseen benchmark suite of 100 test cases spanning 33 distinct real-world domains (Space, Medicine, Biology, History, Physics, Computing, Economics, Law, Myths, and Conspiracies) with a strictly balanced 50 True / 50 False distribution.

1. Overall Performance Metrics

Evaluation Metric	Score	Explanation
Total Test Cases	100	50 Ground Truth True, 50 Ground Truth False
Accuracy	89.00%	89 / 100 correct classifications
F1-Score	0.8991	Harmonic mean of precision and recall
Recall (Sensitivity)	98.00%	49 / 50 factual claims correctly identified
Specificity (TNR)	80.00%	40 / 50 falsehoods successfully rejected
Precision	83.05%	Positive predictive value for True
Negative Predictive Value	97.56%	Predictive reliability when outputting False
Balanced Accuracy	89.00%	Mean of Sensitivity (98%) and Specificity (80%)
Matthews Correlation (MCC)	0.7930	Strong correlation with ground truth

2. Confusion Matrix

                      PREDICTED TRUE      PREDICTED FALSE
ACTUAL TRUE (50)   │    TP = 49        │     FN = 1         │  (Recall: 98.0%)
ACTUAL FALSE (50)  │    FP = 10        │     TN = 40        │  (Specificity: 80.0%)

3. Inference Latency & Hardware Profiling (Apple Silicon Metal GPU)

Metric	Measurement
Mean Latency per Claim	347.04 ms
Median Latency ($p_{50}$)	357.18 ms
90th Percentile ($p_{90}$)	378.82 ms
95th Percentile ($p_{95}$)	393.05 ms
99th Percentile ($p_{99}$)	436.98 ms
Inference Throughput	2.88 claims / second
Total 100-Sample Test Time	34.72 seconds

4. Domain & Category Slice Highlights

• 100% Accuracy Domains: Space Exploration & Astronomy (8/8), Physics (6/6), Archaeology & History (6/6), Zoology & Animal Science (5/5), Geography (3/3), Genetics & Biology (3/3), Medicine & Pharmacology (3/3), Environmental Science (2/2).
• Challenging Domains: Common popular internet myths and folk science (e.g., Einstein failing math, 80% body heat lost through head), where pre-training frequency creates a slight credulity bias toward plausible-sounding claims.

---

Quickstart & Usage

1. Installation

Install the MLX framework:

pip install mlx-lm

2. Python Inference (MLX)

from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler

# Load the standalone fused model directly from Hugging Face
model_id = "ahmednawaz8813/FineTunedLlama3"
model, tokenizer = load(model_id)

# Format the prompt
system_prompt = (
    "You are an expert fact-checking verification system. "
    "Your task is to determine whether a given claim is True or False. "
    "Analyze the claim in the context of the speaker and the setting in which "
    "it was made. Respond with exactly one word: True or False."
)

claim = "The James Webb Space Telescope operates at the Sun-Earth L2 Lagrange point."
speaker = "NASA Astrophysics Division"
context = "Official mission status documentation"

messages = [
    {"role": "system", "content": system_prompt},
    {"role": "user", "content": f"Claim: {claim}\nSpeaker: {speaker}\nContext: {context}"},
]

prompt = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

# Generate single-token deterministic prediction
response = generate(
    model,
    tokenizer,
    prompt=prompt,
    max_tokens=5,
    sampler=make_sampler(0.0),  # Greedy decoding
    verbose=False,
)

print("Verdict:", response.strip())  # Output: True

---

Limitations & Intended Use

• Intended Use: Automated pre-screening and fact-checking of short factual claims, journalistic verification assistance, and educational misinformation detection.
• Out of Scope: Legal advice, clinical medical diagnosis, or unverified breaking news where real-time retrieval-augmented generation (RAG) is necessary.
• Known Limitations: Claims relying on obscure internet folklore myths or fine numerical distinctions may occasionally trigger false positives without retrieval grounding. Pair with a retrieval system (e.g., Wikipedia/Google Search RAG) for production deployment.

---

Citation & References

@inproceedings{wang-2017-liar,
    title = {"Liar, Liar Pants on Fire": A New Benchmark Dataset for Fake News Detection},
    author = {Wang, William Yang},
    booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics},
    year = {2017},
    pages = {422--426}
}

@article{dubey2024llama,
    title = {The Llama 3 Herd of Models},
    author = {Dubey, Abhimanyu and Jauhri, Abhinav and others},
    journal = {arXiv preprint arXiv:2407.21783},
    year = {2024}
}