FCP-360M

Fact-Conditional Pretraining (FCP) makes factual information explicitly available during pretraining by conditioning on triples extracted from each document and predicting the original text. This reduces the pressure to memorize factual associations in model parameters.

This repository contains the final pretraining checkpoint at step 97,560, before downstream task-specific fine-tuning. The model was initialized from scratch using the SmolLM2-360M architecture and tokenizer. Model weights are stored in BF16 safetensors format. The tokenizer includes the factual-triple markers and the training ChatML template.

The processed pretraining corpus is available at anonymous-review-0/FCP-pretrained-20B.

Evidence-conditioned generation

Provide a passage and a question. The included pipeline extracts triples with Qwen/Qwen3-8B, then uses them to condition FCP's answer. The extractor is a separate model and requires additional memory.

Download the inference helpers into a working directory:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="anonymous-review-0/FCP-360M",
    local_dir="fcp_helpers",
    allow_patterns=["fcp_inference.py", "triple_extraction.py", "requirements.txt"],
)

Install the dependencies using Python 3.11:

python -m pip install -r fcp_helpers/requirements.txt

Run from the same working directory:

import sys
sys.path.insert(0, "fcp_helpers")
from fcp_inference import FCPPipeline

pipe = FCPPipeline.from_pretrained(
    "anonymous-review-0/FCP-360M", device_map="auto", torch_dtype="auto"
)
answer = pipe(
    [{"role": "user", "content": "Who wrote Nineteen Eighty-Four?"}],
    context="Nineteen Eighty-Four was written by George Orwell and published in 1949.",
)
print(answer)

Standard Transformers loading

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("anonymous-review-0/FCP-360M")
model = AutoModelForCausalLM.from_pretrained(
    "anonymous-review-0/FCP-360M", device_map="auto", torch_dtype="auto"
)

The helper Python files are explicit local utilities; loading the weights does not require trust_remote_code=True. manifest.json records the released file hashes. Optimizer states, training logs, and private provenance are excluded.

Third-party resources

Architecture and tokenizer: HuggingFaceTB/SmolLM2-360M. Triple extractor: Qwen/Qwen3-8B. The inference helper code is covered by the included LICENSE-code. Third-party resources retain their original terms and attribution.