Lilavati-Checker

A 95M-parameter model that reads a maths word problem, a Python program written by Lilavati and the program's result, and estimates how likely the result is to be right. Lilavati uses it to choose between many tries.

More: Lilavati · try it in your browser · code and evaluation on GitHub

Developer	Udgam Labs
Parameters	94.7M (the same architecture and tokenizer as Lilavati)
Input	a question, one candidate program and its result
Output	the probability that the result is correct
Licence	Apache-2.0

How it is used

Lilavati writes several programs for a question and runs each one. The checker rates every program. Each candidate answer gets the sum of the ratings of the programs that produced it, and the answer with the most support wins. This is what Lilavati.solve() does by default:

from lilavati import Lilavati
lv = Lilavati()                                  # loads Lilavati and Lilavati-Checker
r = lv.solve("Meera deposits 25,000 rupees at 8% simple interest per year. How much interest does she earn in 3 years?", tries=8)
print(r["answer"], r["confidence"])              # 6000, and the checker-weighted share of support for it
print(r["scores"])                               # the checker's rating of each of the 8 programs

Using the model directly. Build this text (with <bos> first), run the model, and compare the next-token logits for " yes" and " no" at the last position. P(correct) is the softmax of those two logits.

<bos>Question: {question}
Answer (Python):
```python
{program}
```
Result: {result, or "error" if the program failed}
Correct:

Keep the last 768 tokens if the text is longer.

Evaluation

GSM8K test (1,319 problems). Lilavati wrote 48 programs per problem at temperature 0.7, and each program was run.

How the answer is chosen	16 tries	48 tries
Checker-weighted vote	68.8	72.3
The checker's single top-rated program	65.9	66.7
Plain majority vote (no checker)	–	65.7
At least one program is right (an upper bound)	–	89.5

Lilavati alone, one greedy try: 54.7.

For comparison, TinyGSM (Liu et al., 2023, arXiv:2312.09241) reports 68.9% when its 125M verifier picks one of 48 programs from its 125M generator. That generator was first pretrained on Phi-1.5 data, and neither model's weights were released. Under that same protocol (the checker's single top pick), Lilavati with this checker scores 66.7%.

Limitations

• It was trained only on training-set problems (no test-set problems) and only on programs in Lilavati's style. Its ratings of programs from other models are untested.
• It can be confidently wrong, especially when most tries share the same mistake, such as using a number that should have been ignored.
• It judges grade-school maths word problems. It is not a general code or proof verifier.
• A high rating is not a guarantee. Check answers that matter.

Licence

Weights: Apache-2.0.

Citation

@misc{lilavati2026,
  title  = {Lilavati: a 95M pocket mathematician that solves word problems by writing Python},
  author = {Udgam Labs},
  year   = {2026},
  url    = {https://huggingface.co/UdgamLabs/Lilavati}
}