ChronoCalc-4B: Calendar Arithmetic Where Calendars Actually Break
ChronoCalc-4B is a LoRA adaptation of google/gemma-3-4b-it for the date and time computations
that real systems get wrong: business days across federal holidays, rolling a date off a weekend,
elapsed time across a daylight saving transition, the next occurrence of a recurring schedule, and
ISO week numbers including the years that have a week 53.
Built for the Adaption AutoScientist Challenge, Part 2 (Math and Code); the training corpus was co-optimized with Adaptive Data (Adaption Labs).
Read this first: what kind of number follows
Adapters are usually scored by win rate, the share of rows where a language model judge preferred the tuned output. Every number on this card is exact-match accuracy against gold answers re-derived from row parameters in stdlib Python, by two independently written solvers that agree on all 60,000 rows.
The two are not interchangeable, and this project has the receipt. A sibling entry in this portfolio trained its identical corpus on Adaption's own AutoScientist. The platform reported a 95.48% win rate. Our scorer, run on that same adapter, measured 62.0% exact-match accuracy. A win rate is a preference; an accuracy is a fact.
| what is measured | exact match against answers computed twice, by two solvers |
| solver disagreements across 60,000 rows | 0, and a disagreeing row is dropped, never repaired |
| a language model judged something | never, not the labels, not the scoring |
| held-out rows scored | 1,150, across four independent slices |
| significance | p = 7.2e-113, exact McNemar, headline slice |
| cost to recompute every number | one laptop, no GPU, stdlib Python only |
And one number this card volunteers that a delta-maximising card would bury:
This model does not claim to take the base from zero. On plain day-walking with no holiday in
the window, gemma-3-4b-it already scores 69 to 86% at every chain depth we tested. Saying
otherwise would be easy and false. The significance test agrees, independently: this base wins
13 held-out rows outright, the only base in this portfolio that wins any at all.
What the base cannot do is handle the exceptions:
| the base model, measured | score |
|---|---|
| plain day-walking, no holiday in the window | 69 to 86% |
| windows where a holiday changes the answer | 7% (2 of 30) |
| elapsed time across a DST transition | 2% |
| cross-year ISO week numbers | 0% |
That is the whole thesis. Deadlines do not go wrong because someone cannot add seven days. They go wrong because Veterans Day fell in the window, or the clocks moved, or the year had 53 weeks.
So the claim here is scoped: improvement on calendar exceptions, and no regression on plain day-walking. Both are reported.
Headline results
Base and tuned generated in one process under identical greedy decoding, scored by the released
score_chrono.py, which re-derives all 60,000 gold answers from row parameters in stdlib Python.
| slice | rows | base | ChronoCalc-4B |
|---|---|---|---|
| headline held-out | 500 | 11.0% | 91.0% |
| hard shard | 300 | 14.7% | 93.3% |
| enumerated calendar | 200 | 23.5% | 90.5% |
| range slice, 2031 to 2035 | 150 | 14.7% | 76.7% |
Significance, stated rather than assumed
Paired design, so significance is an exact McNemar test on the rows where exactly one model was right, with the null that the adapter is no better than the base.
| slice | only tuned right | only base right | exact two-sided p |
|---|---|---|---|
| headline held-out | 405 | 5 | 7.2e-113 |
| hard shard | 241 | 5 | 1.3e-64 |
| enumerated calendar | 135 | 1 | 3.1e-39 |
| range slice, 2031 to 2035 | 95 | 2 | 6.0e-26 |
Note the middle column, which is not zero here. Alone in this portfolio, this base wins rows outright: 13 of them across the four slices. That is the same fact the top of this card opens with, arriving independently from the test rather than from the prose. A base that scores 69 to 86% on plain day-walking should win some rows, and it does.
Recompute with the released significance.py from the released generation files.
The gate that nearly killed this entry, and the rule written before it ran
The pre-registered Day-0 gate measured a blended F1 floor of 34%, which sat between the pass and
kill bands. The decomposition explains why: the blend is a property of the pilot mix, not of the
model. The pilot was 60% holiday-affected, and share x 7% + (1 - share) x 70% reproduces the
measured 34% to within half a point.
Because "between the bands after a rerun" was undefined in the spec, an adjudication rule was
written and committed to git before the rerun was generated or scored
(docs/eval/gcp/chrono_gate_adjudication.md). An adjudication rule invented after seeing the number
it adjudicates is worthless. The rule set four conditions, all of which had to hold or the entry
died with no further branches. It passed.
This is on the card as measurement transparency rather than buried, in the same way the DataViz entry shipped its disclosed regression with a diagnosis.
Every answer was computed twice
solver_a produces the answer and the working. solver_b, written from scratch with different
primitives, produces the answer again. A row whose solvers disagree is dropped and logged, never
repaired. There are 0 disagreements across 60,000 rows, and no language model appears anywhere
in the label path.
A reviewer can recompute every published number on a laptop with no GPU.
Scope and operating notes
- The base is not bad at dates in general, and this card refuses to imply otherwise. The improvement is scoped to calendar exceptions.
- 90% of rows name the holiday calendar by statute rather than enumerating it, because knowing the US federal calendar is part of the trained behaviour. The 200-row enumerated held-out slice prints the calendar in full, so the headline can be rechecked without any statute knowledge. Report both.
- Scope. US federal holidays, Gregorian dates, the IANA timezone database. Not other national calendars, not historical calendar reforms, not leap seconds.
- Completions are long on purpose. Every one enumerates its computation line by line before the
<answer>tag. A terser format would score the same on exact match and be worth less, and terseness has twice generalized out of a shard on this project and cost win rate on untouched tasks.
Training details
| base | google/gemma-3-4b-it |
| method | LoRA r32 alpha64 on the language-model linears, completion-only masking |
| corpus | 60,000 rows, 5 families, 51 distinct templates |
| epochs | 3 |
| max_len | 1024, measured (total p99 833, max 960, 0 of 60,000 over budget) |
| hardware | one A100-40GB, Spot |
max_len is measured because a 768 window would truncate roughly the top decile, and a truncated
completion loses its <answer> tag entirely, so the row would teach nothing under completion-only
masking.
Evaluation protocol
Base and tuned generated in one process under identical greedy decoding (do_sample=False).
Output length compared base against tuned, with a tuned median below 0.6x the base median failing the
run regardless of accuracy. Every gate was proven able to fail by injecting deliberately corrupted
rows before the corpus was trusted.
Usage
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
BASE = "google/gemma-3-4b-it"
tok = AutoTokenizer.from_pretrained(BASE)
# AutoModelForCausalLM resolves gemma-3-4b-it to its multimodal wrapper, which is correct.
# Do NOT load it through a text-only class: the decoder is then randomly initialized and
# the model emits whitespace, with no error raised.
model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype="bfloat16",
attn_implementation="eager", device_map="cuda")
model = PeftModel.from_pretrained(model, "Jainamshahhh/chronocalc-4b").eval()
Try it, and everything that backs it
Live side by side demo: https://huggingface.co/spaces/Jainamshahhh/chronocalc-demo Enter your own input and watch the base model and ChronoCalc-4B answer it under identical greedy decoding. The GPU backend scales to zero, so a cold first request takes about a minute.
Released on both platforms, with the scorer, every eval slice, the per row verdicts and
significance.py alongside, so every number on this page can be recomputed rather than
trusted.
Reproducibility and license
The corpus regenerates byte for byte from a single seed, because a row is a pure function of its integer id. The scorer, every held-out slice and both solvers are published. Apache-2.0, matching the base model. Built with Adaptive Data by Adaption Labs, whose enhancement pass on this corpus was run, measured, and refused when it rewrote dates inside the questions; that decision is documented rather than omitted.
- Downloads last month
- -