T-CRED-SL: lightweight semantic evaluation for Temporal Graph RAG

T-CRED-SL (T-CRED Semantic-Learned) is a 33.4M-parameter English cross-encoder for fine-grained evaluation of retrieval-augmented generation, Temporal RAG, Graph RAG, temporal question answering, and temporal knowledge-graph QA outputs. It scores six semantic constructs: answer correctness/equivalence, evidence support, retrieval relevance, temporal support expressed in language, answerability, and citation appropriateness.

This is an evaluation model, not a question-answering or text-generation model. It is also a deliberately preserved negative-result research artifact: on the project's final human and source-disjoint meta-evaluation, this learned model did not outperform deterministic T-CRED v1.4. It is released as a lightweight baseline, an auditable multi-task checkpoint, and a starting point for research on learned RAG metrics. It must not be described as a superior replacement for T-CRED v1.4 or for human evaluation.

Why this model exists

Common answer-overlap and generic factual-consistency metrics do not separately diagnose whether a RAG answer is semantically correct, supported by retrieved evidence, valid at the requested time, appropriately cited, or answerable from the available context. T-CRED-SL tests whether a compact shared encoder can learn the semantic parts of that measurement problem while exact interval, directed-path, provenance, and missingness rules remain deterministic.

The experiment produced an important negative finding: strong held-out semantic-task performance was not sufficient to preserve construct separation, temporal sensitivity, graph validity, missingness behavior, calibration, and update invariance in the final metric suite.

Intended uses

• research on automatic RAG evaluation and learned evaluation metrics;
• semantic scoring of candidate answers, claims, evidence passages, and citations;
• lightweight baselines for Temporal RAG, Graph RAG, temporal QA, and grounded generation;
• ablation studies comparing learned semantic judgments with deterministic temporal/graph rules;
• reproduction of the reported T-CRED-SL negative result.

Do not use it as

• a standalone truth oracle or substitute for expert human judgment;
• a generative QA model, retriever, reranker, or temporal knowledge-base reasoner;
• a complete implementation of deterministic T-CRED v1.4;
• a metric for exact timestamp arithmetic, directed graph-path validity, provenance availability, or required-citation missingness;
• a commercially deployable model. The checkpoint is restricted to non-commercial research because its training mixture includes ANLI and MS MARCO. See LICENSE.md.

Model at a glance

Property	Value
Author	Murad Mustafayev
Hosting organization	Quicksort-fr
Backbone	MiniLM-L12-H384-uncased, full fine-tuning
Parameters	33,371,536
Layers / hidden size / attention heads	12 / 384 / 12
Task families	6
Prediction heads	8 linear outputs across the 6 tasks
Maximum input length	256 wordpiece tokens
Tokenizer size	30,536, including 14 task/field tokens
Language	English only
Training rows	798,565
Development rows	71,033
Calibration rows	17,854
Training presentations	1,097,111
Optimizer steps	8,589
Final seed	42; one completed final seed
Weight format	SafeTensors
Weight SHA-256	33e2e19437aaea4b7ebadf3e3fc2102cf700197ece3cff31b9fc6d5fee5ca81c

Quick start

Install the runtime:

pip install "torch>=2.6,<3" "transformers>=4.53,<5" safetensors

Load the checkpoint and run calibrated inference. trust_remote_code=True is required for the custom six-head architecture; review modeling_tcred_sl.py and pin a commit revision in production.

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "Quicksort-fr/T-CRED-SL"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True)
model.to("cuda" if torch.cuda.is_available() else "cpu")

records = [
    {
        "input_id": "a-1",
        "task": "answer",
        "question": "Who led the organization in 2020?",
        "query_time_or_interval": "2020",
        "temporal_operator": "during",
        "reference_answers": ["Alex Morgan"],
        "candidate_or_claim": "Alex Morgan",
    },
    {
        "input_id": "s-1",
        "task": "support",
        "question": "Who led the organization in 2020?",
        "candidate_or_claim": "Alex Morgan led the organization in 2020.",
        "evidence_passages": [
            {
                "evidence_id": "e1",
                "text": "Alex Morgan served as director from 2019 through 2021.",
            }
        ],
    },
]

predictions = model.predict(tokenizer, records, batch_size=32)
print(predictions)

The repository also contains an executable example.py. See USAGE.md for the complete input contract, all six task examples, batching, output interpretation, and raw-head inference.

Tasks and outputs

Task	Required semantic content	Calibrated outputs
answer	question, candidate; normally references	answer_u1, answer_u2, equivalence, score
support	claim/candidate and evidence	entailment/neutral/contradiction distribution; supported
relevance	question and evidence	relevance
temporal	claim/evidence plus time or temporal operator when available	support/unknown/contradiction distribution; supported
answerability	question and available evidence/context	answerable
citation	question, candidate, evidence set, cited evidence IDs	appropriate/incomplete/inappropriate distribution

For the ordinal answer head:

• answer_u1 = P(answer is at least partially correct);
• answer_u2 = P(answer is fully correct), constrained so answer_u2 <= answer_u1;
• score = (answer_u1 + answer_u2) / 2;
• equivalence is a separate question-conditioned reference-equivalence probability.

Scores are model probabilities on the frozen calibration scale. They are not guarantees of factual truth, and they should not be averaged across constructs without a separately justified decision rule.

Architecture

The model uses a shared MiniLM-L12-H384 encoder. The final [CLS] representation feeds:

1. a three-logit ordinal answer/equivalence head;
2. a three-class support head and a separate binary support head;
3. a binary relevance head;
4. a three-class temporal head and a separate binary temporal-support head;
5. a binary answerability head;
6. a three-class citation-quality head.

Inputs are serialized with frozen task and field tokens. Dataset identity, split identity, label provenance, and target labels are never model inputs. Per-head scalar temperature scaling is fitted only on the held-out calibration partition and is embedded in config.json.

See ARCHITECTURE.md for equations, serialization order, and head semantics.

Training data

The selected corpus contains 887,452 unique English model inputs: 798,565 training, 71,033 development, and 17,854 calibration rows. It combines MultiNLI, WANLI, ANLI, PAWS-Wiki, FEVER, VitaminC, public-license Temporal NLI sources, MOCHA, Answer Equivalence, AttributionBench, RAGTruth, TORQUE, SQuAD 2.0, MS MARCO v1.1, and fresh source-disjoint project-formal examples backed by Wikidata-derived scenarios.

Partitions are group-disjoint. Exact duplicates were removed, conflicting identical inputs were quarantined, and near-duplicate selection used MinHash only for candidate generation followed by an exact word-trigram Jaccard decision at 0.90. The current project human gold, released benchmark units, retrospective test, and source-disjoint formal challenge were excluded from training.

No training examples are redistributed in this model repository. Exact source revisions, selected counts, adapters, split policy, and terms are in TRAINING_DATA.md and metadata/license_ledger.json.

Training procedure

• Stage A: 500,019 broad semantic rows, one epoch.
• Stage B: 298,546 task-matched rows, two epochs.
• AdamW, learning rate 3e-5, weight decay 0.01, effective/micro batch size 128.
• BF16 on one NVIDIA A100 80GB; maximum length 256; dropout 0.10.
• 6% warmup (515 steps), gradient clipping at 1.0.
• Task loss plus weighted pairwise margin (0.40), invariance (0.20), and Brier (0.05) terms.
• Frozen seed 42; 8,589 optimizer steps; 1,811.612 seconds measured training time.
• Held-out scalar temperature calibration by head after training.

See TRAINING.md and the immutable files under metadata/ for full details.

Evaluation results

Held-out semantic calibration partition

These task-level results show that the network learned much of its direct supervision. They do not establish that it is a valid end-to-end temporal graph RAG metric.

Output	n	Result
Answer u1	997	AUROC 0.9559; macro-F1 0.8786
Answer u2	1,177	AUROC 0.9568; macro-F1 0.8790
Answerability	1,267	AUROC approximately 1.0000; macro-F1 1.0000
Answer equivalence	1,444	AUROC 0.9245; macro-F1 0.6630
Relevance	1,426	AUROC 0.8313; macro-F1 0.7374
Binary evidence support	817	AUROC 0.9647; macro-F1 0.9344
Binary temporal support	2,265	AUROC 0.8876; macro-F1 0.7915
Citation class	701	accuracy 0.9001; macro-F1 0.9166
Support class	8,729	accuracy 0.7139; macro-F1 0.7032
Temporal class	1,872	accuracy 0.9551; macro-F1 0.8365
Scalar answer rating	622	MAE 0.2005; Spearman 0.7323
Controlled ranking	1,294 pairs	accuracy 0.9119

Final meta-evaluation against T-CRED v1.4

The replacement hypothesis was rejected. On the 113-unit provisional human gold set, T-CRED-SL had lower Spearman correlation than deterministic T-CRED v1.4 for every reported construct. The largest differences were answer correctness (0.667 vs 0.915) and graph sufficiency (0.285 vs 0.767). T-CRED-SL's evidence-support AUROC was higher (0.979 vs 0.909), but its Spearman correlation and calibration were worse (MAE 0.472 vs 0.195).

On 4,088 source-disjoint formal cases containing 5,427 matched pairs, T-CRED-SL was worse on most directional and invariance tests. Recurring failures included wrong-direction answer changes after snapshot updates, missing-citation handling, ties on temporally invalid retrieval metadata, invalid graph-path sensitivity, and presentation-order sensitivity.

No human field significantly favored T-CRED-SL. Its small formal temporal-attribution gain was not statistically significant. The checkpoint is therefore retained as a baseline and negative result, not promoted as the primary metric.

See EVALUATION.md for complete tables, confidence intervals, controlled failures, runtime, and interpretation. Machine-readable reports are under metadata/.

Limitations

• English only.
• One completed final seed; no multi-seed uncertainty estimate.
• Human validation is exploratory: 113 gold units, sparse field-wise labels, low-to-moderate initial agreement, and substantial adjudication.
• The checkpoint can conflate semantic support with temporal correctness.
• The fixed 256-token limit truncated 4.775% of selected corpus rows, concentrated in Stage B.
• Calibration is distribution-specific and does not repair construct mismatch or domain shift.
• A semantic model cannot reliably replace exact interval algebra, directed path validation, provenance checks, or deterministic missingness rules.
• Scores may shift with evidence order and top-k selection.
• Training-source licensing limits use to non-commercial research.

See LIMITATIONS_AND_ETHICS.md for the complete limitations, bias, privacy, environmental, and misuse statement.

Files

Path	Purpose
model.safetensors	complete encoder and all task-head weights
config.json	architecture, calibration, frozen model metadata, AutoModel mapping
tokenizer files	frozen 30,536-token tokenizer
configuration_tcred_sl.py	custom Transformers configuration
modeling_tcred_sl.py	model, exact formatter, calibrated batched inference
example.py	executable quick start
metadata/	manifests, development/calibration reports, logs, source ledger, checksums
research/	original design, protocol, and final evaluation reports

Reproducibility and integrity

The uploaded model.safetensors is byte-identical to the independently validated final export. Use SHA256SUMS to verify the complete publication bundle. The checkpoint identity is:

33e2e19437aaea4b7ebadf3e3fc2102cf700197ece3cff31b9fc6d5fee5ca81c  model.safetensors

The frozen run manifest reports 8,589/8,589 completed steps and training configuration hash f52a7483d9bedd69b6d1c6fa042c0d70b71531dc5fc197d0a2cc0743fa5bf157.

License and terms

license: other is intentional. The backbone is MIT-licensed, but the training mixture includes ANLI (CC BY-NC 4.0) and MS MARCO (non-commercial research terms), plus sources with attribution and share-alike conditions. This checkpoint is provided for non-commercial research use only. No training text is redistributed. Review LICENSE.md, NOTICE.md, and every upstream source's current terms before use. This is a provenance notice, not legal advice.

Citation

Until a corresponding archival paper is available, cite the model artifact and the MiniLM backbone:

@misc{mustafayev2026tcredsl,
  title        = {T-CRED-SL: A Lightweight Learned Semantic Metric for Temporal Graph RAG},
  author       = {Mustafayev, Murad},
  year         = {2026},
  howpublished = {Hugging Face model repository},
  url          = {https://huggingface.co/Quicksort-fr/T-CRED-SL},
  note         = {Version 1, MiniLM-L12-H384, seed 42}
}

Core architectural reference:

Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers. NeurIPS.

The full source-paper bibliography is in REFERENCES.md.