OVRLab/granite-jev-1b-sampling

Granite–Jev: sampling + selective verification

A runnable inference procedure, using unchanged IBM Granite 4.0–1B weights and TypeSafe's hosted Jev verifier. This release implements the current sampling-only paper. Research code.

There are no newly trained weights, adapters, internal routing changes or Jev weights in this package. It is a custom serial Python runner, not an AutoModel checkpoint, hosted Hugging Face chat widget or vLLM plugin. The earlier trained-proposal artifact is a different preserved experiment.

What runs

1. Frozen Granite generates one greedy answer.
2. The same weights generate one independent sample (temperature 0.7, top-p 0.95, top-k 50), with a separate cache and a reproducible input-specific seed.
3. Exactly equal texts keep the native answer without calling Jev. Otherwise Jev 1.13.0 scores the anonymous pair against the visible task, without references.
4. Keep the sampled answer only if its score exceeds native by strictly more than 0.05. Return exactly that branch's generated token IDs.

Jev runs during inference and sees the prompt and both candidate texts. Both Granite generations are paid for even when verification is skipped. Jev does not generate the answer tokens. Its scores are fallible and are not established as calibrated correctness probabilities.

Download and run

Use Python 3.12 (validated environment); pinned PyTorch 2.8.0 and Transformers 4.57.1. First use the version-pinned download instructions, which give the exact verified Hugging Face commit. Then:

cd granite-jev-sampling
python -m venv .venv
. .venv/bin/activate
pip install -r requirements.txt
python run_sampling.py --verify
python run_sampling.py --input example.json --mode native --device cpu --max-new-tokens 128

The base weights/tokenizer are downloaded directly from IBM at revision 6a7381ba1f54d684ff508d991aeb7dc580157103. CPU execution is supported but slower; use --device cuda on a suitable GPU. The runner uses FP32. It does not silently quantize, truncate the prompt or change precision to fit a device.

For Jev selection, put your own raw TypeSafe key in a private file and run:

python run_sampling.py --input example.json --device cuda \
  --key-file ~/.typesafe.ai/jev --output answer.json

Alternatively supply TYPESAFE_API_KEY through your runtime's secret mechanism. An explicit key file takes precedence; the default file is ~/.typesafe.ai/jev. Native mode and artifact verification need no key. No key is stored in weights, configuration, prompts or the published artifact. Hosted use incurs the provider's charges and requires access to the pinned Jev model.

Guided CLI runs require --output. The CLI exclusively reserves this new file before model loading or any provider call, so two commands cannot spend on the same destination. It initially records in_progress and retains an explicit failure if inference fails. Input and verifier-payload hashes are flushed to disk before dispatch and retained on failure; unknown charges remain explicit. Missing credentials and expired pre-dispatch deadlines are recorded as uncharged. Records update by atomic replacement with file/directory synchronization, so an interrupted update preserves the previous record. Reported usage and bounded, redacted provider diagnostics survive invalid responses. Successful results identify the loaded model revision, artifact digest, device, precision and runtime versions. Inspect a leftover reservation before retrying; it is not a successful answer. This is not a transactional job service or an automatic paid-request recovery mechanism.

Input and limits

{"id":"my-question-1","task":"math","prompt":"What is 9 + 6? Give just the number."}

Allowed task rubrics: instruction, math, reasoning, code, code_output, safety, grounding. These names choose verification instructions; they are not assertions that a question belongs to an independently validated benchmark. Include all relevant visible source material in prompt. Extra fields such as reference answers are rejected. An omitted ID is derived from the input. Saved results include the task rubric, stable ID and canonical input hash; guided judgments also record a hash of the exact verifier payload. Keep the input and pinned release revision with the result so the judgment contract can be audited.

--replicate chooses 3701, 3702 or 3703; each is a separate sample-and-verification draw, not best-of-three. Outputs are capped at 2,048 tokens per branch and total context at 32,768 tokens. The default output cap is 1,024. --max-seconds is a bounded request deadline. One runner owns one serial request; no generated code is executed. A user prompt is sent to Jev only on disagreement. The API is synchronous: invoke it outside a running asyncio event loop. An event-loop rejection happens before provider dispatch and records zero API attempts.

Evidence and reproducibility

The paper currently reports thirteen broader endpoints from eleven generation cohorts, with absolute gains and paired uncertainty. They are exploratory saved experiments, not a guarantee for arbitrary prompts. This demo is not the full benchmark harness: historical few-shot chat messages, task-specific parsers, stopping limits and reasoning choice-agreement gates remain cohort-specific. Use the paper's protocols and pinned source for those experiments.

source_manifest.json identifies the frozen functions used for the generation path. VALIDATION.md distinguishes offline tests, real-model parity and live provider evidence. artifact_manifest.json detects accidental file changes; it is not a cryptographic signature. Pin the Hugging Face commit for repeatable downloads. Scores may vary across devices/provider executions.

The demo makes at most one provider attempt per request. A failed or invalid judgment raises an explicit error; it is never counted as a successful native fallback. Do not blindly rerun a timed-out paid call. Benchmark campaigns use a separate durable journal and bounded five-minute outage recovery; this small runner does not claim that recovery behavior or server-grade persistence.

License and limitations

OVRLab code is MIT; IBM base weights are separately Apache 2.0. Jev is a hosted service with separate terms, availability and charges. No Jev weights or benchmark text are redistributed. See NOTICE.md and SECURITY.md.

This system can produce incorrect answers and harmful corrections. It adds generation and API overhead. Current evidence does not establish universal improvement, superiority at equal resources, a new internal LLM architecture, or replacement of a larger model. No endorsement by IBM or TypeSafe is implied.

If a failed response contains only one valid token counter, the failure record preserves it in jev_usage and names missing/invalid counters in usage_unknown_fields; those partial counts are not a complete bill.

The durable CLI supports Linux and macOS. Native Windows is outside this support boundary; use a Linux environment such as WSL2. Unsupported platforms fail before file reservation or model loading; --verify remains available. If saving a completed result fails, a successful fallback record retains the exact answer and known usage with inference_complete: true and failure_stage: result_persistence; it must not trigger another paid request.

If the attempted-payload record cannot be saved, Jev is not called: the failure records zero attempts and no unknown provider charge.