Title: Grammars of Formal Uncertainty: When to Trust LLMs in Automated Reasoning Tasks
Venue: NeurIPS 2025
Accepted: yes
Keywords: Large Language Models

Formal Verification

Uncertainty Quantification

Stochastic Context-Free Grammars

Automated Reasoning

SMT-based Autoformalization

Selective Verification
Abstract: Large language models (LLMs) show remarkable promise for democratizing automated reasoning by generating formal specifications. However, a fundamental tension exists: LLMs are probabilistic, while formal verification demands deterministic guarantees. This paper addresses this epistemological gap by comprehensively investigating failure modes and uncertainty quantification (UQ) in LLM-generated formal artifacts. Our systematic evaluation of five frontier LLMs reveals Satisfiability Modulo Theories (SMT) based autoformalization's domain-specific impact on accuracy (from +34.8\% on logical tasks to -44.5\% on factual ones), with known UQ techniques like the entropy of token probabilities failing to identify these errors. We introduce a probabilistic context-free grammar (PCFG) framework to model LLM outputs, yielding a refined uncertainty taxonomy. We find uncertainty signals are task-dependent (e.g., grammar entropy for logic, AUROC>0.93). Finally, a lightweight fusion of these signals enables selective verification, drastically reducing errors (14-100\%) with minimal abstention, transforming LLM-driven formalization into a reliable engineering discipline.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates how to quantify and manage formal-logic uncertainty that arises when large language models (LLMs) automatically translate natural-language problems into symbolic formats (specifically SMT-LIB). \nThe authors diagnose failure modes by benchmarking five state-of-the-art LLMs on four reasoning datasets, showing that symbolic translation can both help and hurt depending on task structure.\nThey also model the space of LLM-generated programs with PCFG, treating each generated SMT string as a sample from an (unknown) distribution and fitting a PCFG to recover rule-level probability mass.\nThey introduce a taxonomy of 25 uncertainty signals derived from the fitted grammar (entropy, spectral radius, rule sparsity, self-consistency, etc.) and show that different tasks prefer different signals.\nThey also propose a lightweight fusion strategy that combines the best signals into a model-agnostic selective-verification policy, reducing downstream logical errors by 14 – 100 % while abstaining on only a small fraction of cases.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths \n- Comprehensive evaluation across multiple models, tasks and metrics; theory and experiments align\n- Elevates uncertainty estimation in neurosymbolic pipelines; practical recipe for safer auto-formalisation\n- Novel use of fitted PCFGs to harvest uncertainty features; fusion policy yields state-of-the-art selective-verification gain\n\nWeaknesses\n- The paper should be clearer. For example, figure 1 is not very clear. Explain more about the colors. Better split the pie chart. \n- No runtime/memory profile for PCFG extraction\n- Lack of scalability/extensibility discussion\n\nQUESTIONS:\nHave you tried the pipeline on a markedly different formal language (e.g., Coq proofs)? Even a small pilot study would strengthen the claim of broad applicability.\n\nWith 25 handcrafted signals, how did you avoid overfitting to the chosen datasets? \n\nEngineers need a threshold to decide when to abstain. A decision-curve or cost-benefit analysis would make the selective-verification story more actionable.\n\nLIMITATIONS:\nAdding quantitative overhead measurements and a scalability/extensibility discussion would improve the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces probabilities to the idea of using grammars for modeling LLM outputs, and then employs them as a means for uncertainty quantification (UQ) in SMT generation tasks. The results are demonstrated using o3-mini, DeepSeek-R1, DeepSeek, Gemini Flash 2.0 and Gemini Lite.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. A key strength of the paper is the inclusion of probabilities into the formal grammar being used for further UQ analysis.\n2. The paper seeks to bring formal methods and LLMs together, which is an interesting direction to pursue.\n3. The UQ metrics are obtained from the spectral and information theoretic measures related to the Jacobian and the non-terminals/rules of the grammar, which is interesting.\n\nWeaknesses:\n1. There is a rich body of work that uses formal grammars at inference time without probabilities. The paper does not adequately discuss these approaches or builds the results of this work in context of these earlier methods. \n2. The title \"Grammars of Formal Uncertainty\" is not apt as the paper does not build grammars of formal uncertainty. Similarly, the broad mention of automated reasoning tasks leaves the strong ties that the paper has to SMT methods, which are only one set of automated reasoning methods.\n3. Theorem 1 (first two results) on page 4 is a straightforward result that can be proved directly from the definition of entropy and it can be included in the text of the paper. Appropriate citations should be provided for each of the steps, such as T. M. Cover and J. A. Thomas, Elements of Information Theory.\n4. It is not clear if the probabilities in the PCFG are obtained from a different data set than the UQ measures obtained during analysis. If the train and test set come from the same data set, the UQ methods will not generalize and will be reliant on the data set distribution.\n\nQUESTIONS:\n1. Please put Theorem 1 in context of known results with citations, if possible.\n2. Please clarify if the probabilities for the PCFG are computed from the same data distribution as used for the UQ metrics.\n3. Discuss the novelty in the paper in context of related work on formal grammars and LLMs.\n\nLIMITATIONS:\nThe effiicacy of UQ metrics and risk of incorrect applications may be better discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMany recent studies have focussed on how to exploit LLMs to solve reasoning tasks (for example, basic logical reasoning). When solving such a task, LLMs can be used in at least two ways: (1) to directly reason on textual representation, or (2) to translate to a format that can then be fed into a specialized reasoning tool (also called \"autoformalization\"). The current paper proposes to study the second approach by using LLM to translate a given reasoning task to SMT-LIB, for which many solvers have been developed. That is, after the translation, one may invoke an SMT-solver (e.g. Z3) to solve the reasoning task. Owing to potential ambiguity in texts, the authors proposed to quantify \"uncertainty\" in the translation by getting LLMs to output the top N formalizations of the sentence in SMT-LIB and extrapolate a Probabilistic Context-Free Grammar (PCFG) out of these, based on which CFG rules in the SMT-LIB grammar that were used in generating these formalizations. That is, the authors studied how SMT solution consistency (e.g. the results of Z3 on the 100 formalizations) correlates to ground truth and that compares to textual based reasoning. The authors tested this on four reasoning benchmarks: ProofWriter, FOLIO, StrategyQA and ProntoQA. The results are mixed: on ProofWriter, the SMT-based approach helped, but for FOLIO it performed worse than direct textual-based reasoning. For this reason, the authors developed many (25) uncertainty metrics, based on these they introduced a lightweight uncertainty signal whether an error has been introduced in the autoformalizations. They have systematically evaluated these on the four benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper attacks a highly timely and relevant problem. Turning a reasoning task using LLM into a format that can be handled by a formal verification tool is not new, e.g., in [Wu et al. 2022]. To the best of my knowledge, this is the first time PCFG has been used in the context of quantifying uncertainty of autoformalizations. Finally, based on this, the authors have developed and systematically evaluated multiple uncertainty metrics, based on them signals for telling whether or not the LLM+SMT-solver approach should be trusted. \n\nNow I would like to mention the weaknesses. \n\nPresentation: I found the presentation to be unclear and rather difficult to follow in many places.\n- Some sentences in the introduction I found rather misleading, including \"rigorous standards required for formal reasoning\" since I do not see how the current work addresses this. \n- In approximation (p. 3-4): some of the technical descriptions get quite dense, esp. for non-experts (e.g. spectral radius, Kullback-Leibler, etc.). In particular, Theorem 1 is mentioned there (proof in the appendix), but is never mentioned again in the paper and it's not clear how it ties to the rest of the paper.\n- Some parts of the experiments are full of very long sentences that are very hard to understand (e.g. l 282 to 286 is one full sentence with lots of breaks!!). \n- I also feel that some concrete, usable signal is hard to extract from the paper. It would be good if the authors concentrate on simplifying this signal from the sea of statistical data from Section 3.\n- You mentioned you developed 25 uncertainty metrics. Can you enumerate them?\n- Sec 3.1: the simplest metric (MLE) works best, which seems to defeat the purpose of discussing other metrics.\n\nSome missing data: \n- Since SMT-solvers can be quite expensive, I think the authors should also report on the running time of Z3 on the 100 formulas that LLM produce. In particular, how does it contrast to other approaches and why for some benchmarks the cost of running SMT-solver was claimed to outweigh the benefits?\n- Is there a reason to use N=100? Would it work better if I use N=50 or less?\n- How large is the PCFG that is generated? \n\nSome additional remarks:\n- some shorthands were not defined in the paper (e.g. SC, line 189). Does it mean \"Self-Consistency\"?\n\nQUESTIONS:\nPlease address the questions above \"in Strengths and Weaknesses\". In addition, can you clarify the following?\n- Can it happen that LLM produce the correct translation if you ask for one translation, but because you ask for 100 translations, it might produce a lot of wrong ones? In this case, it seems that the approach puts way more weights on translations that are wrong.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work addresses the problem of uncertainty quantification for LLMs when used to generate formal artifacts like SMT-LIB programs. For this purpose, it introduces a sampling-based method to estimate a probabilistic CFG for the model's underlying distribution for SMT-LIB programs. They also come up with a taxonomy of uncertainty and evaluate how various UQ metrics based on their methodology perform on four on four datasets commonly used for reasoning tasks: StrategyQA, ProntoQA, ProofWriter and FOLIO.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n* I think the problem addressed by the paper is important. There seem to be a few recent works that rely on the LLM formlasing certain logical tasks, and then using some type of solver to get an answer. However, many of the works don't clearly address the issue of the formalization not being accurate. That's a weak spot that may invalidate all the benefits of the subsequent formal reasoning.\n\n* Although prior work (e.g., Li et al., CAV’24) has explored using pCFGs to model LLM outputs (which should be clearer acknowledged in the related works), I think this paper is the first to apply pCFGs specifically for uncertainty estimation.\n\nWeaknesses:\n\n* The approach is limited to syntactic variations between candidates and does not consider semantic equivalence. For instance, the authors should check out the work on semantic uncertainty in Kuhn et al., ICLR'23 and Farquhar et al, Nature'24. This is for me a serious issue with this paper. In this paper, I think that syntactically different but semantically equivalent samples will result in a high grammar entropy. However, intuitively and according to Kuhn at al., that scenario should lead to low entropy.\n\n* Clarity wise, Figure 1 is difficult to read, which undermines its usefulness as an overview. Given the large number of derived metrics introduced in the paper, a better organised by using some sort of abstraction would be helpful. The same issue also made it challenging at times to interpret the results in Tables 3 and 4.\n\nQUESTIONS:\n* Do you have any comments on the lack of a comparison with uncertainty measures considering semantic clustering (e.g., Kuhn et al., ICLR'23)? \n\n* In the Approximations section (lines 94-103) there is a description about how the N samples are used to infer the various production rules for the pCFG. I wonder how reliant this technique is on the quality of samples produced? For instance, would increasing the temperature to get more diverse samples harm the accuracy of the estimated pCFG?\n\n* Just to make sure I got it right, you use 5 samples per question to construct the grammar? Did you try to vary this number, maybe increase it?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}