NagaYu/ante-claim-alignment

Ante — claim-alignment judge

Given a change, the claim made for it, and the evidence attached, this judge answers one question:

Does this evidence actually test what this claim says?

It does not answer "is this patch correct?" — a much harder question that no system answers reliably — and it does not answer "who wrote this?", which it refuses to ask.

This is a rule-based program, not a neural network

That is deliberate. The judge's output can cost a contributor real work, so every decision has to be explainable line by line to the person it affects. It combines four executed or statically-derived signals:

1. Discrimination — the attached test fails on the base commit and passes on head, when re-run independently. A base failure that is an ImportError for a symbol the patch introduces is recorded as weaker than an AssertionError about behaviour.
2. Reach — which changed executable lines the evidence actually ran (line-traced; comments, blank lines and docstrings are excluded from the denominator so honest patches are not penalised). Threshold: min_change_coverage = 0.34.
3. Correspondence — whether the assertions mention the symbols, exception types, keyword arguments and conditions the claim is about.
4. Non-triviality — an AST analysis that recognises tautological assertions, constant-only assertions, tests that never call into the package, and tests that mock out the very symbol under claim. It tracks taint, so the ordinary result = f(x); assert all(... for ... in result) style is correctly read as substantive.

A fault-injection probe (mutation testing restricted to the changed lines) is reported as a signal, not a gate: measured on the benchmark, gating on it costs false positives, because small guard-clause fixes legitimately offer few faults to inject.

Verdicts

ALIGNED · TRIVIAL_EVIDENCE · MISALIGNED · UNDER_SUBSTANTIATED · NO_EVIDENCE

None of them means "rejected". The protocol's negative outcome is a specific, satisfiable request.

Measured on Ante Bench

metric	value
fabricated evidence detected	100%
off-topic / low-quality rejected	100%
good contributions lost (false positives)	6.8%
behaviour-breaking changes accepted	30%
verification time per PR (mean / max)	0.42s / 2.14s

Compare against the baselines in the benchmark.

Limitations, stated plainly

• It verifies the claim that was made, not the claims that were not. A patch whose stated performance claim is true but which silently changes behaviour the project does not test will pass.
• Claim extraction is lexical. A claim written without naming any symbol the repository defines yields a weaker signal.
• Benchmark evidence depends on the machine it runs on; confidence intervals mitigate this but do not remove it.
• Thresholds were chosen on this corpus. They are declared in config.json and overridable per project in AGENTS.md.

Usage

from predict import AlignmentJudge

judge = AlignmentJudge(path="pkg/core.py")
print(judge.judge(claim=issue_body, before=old_source, after=new_source,
                  test_source=attached_test))

Not for authorship detection

This repository contains no authorship signal and must not be used to guess whether a human or a model wrote a contribution. That classification is unreliable and unfair, and the protocol this judge belongs to is built to make it unnecessary.