Title: Teaching Transformers to Solve Combinatorial Problems through Efficient Trial & Error
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

Transformers

Combinatorial

Sudoku

SAT
Abstract: Despite their proficiency in various language tasks, Large Language Models (LLMs) struggle with combinatorial problems like Satisfiability, Traveling Salesman Problem, or even basic arithmetic. We address this gap through a novel trial \& error approach for solving problems in the class NP, where candidate solutions are iteratively generated and efficiently validated using verifiers. We focus on the paradigmatic task of Sudoku and achieve state-of-the-art accuracy (99\%) compared to prior neuro-symbolic approaches. Unlike prior work that used custom architectures,  our method employs a vanilla decoder-only Transformer (GPT-2) without external tools or function calling. Our method integrates imitation learning of simple Sudoku rules with an explicit Depth-First Search (DFS) exploration strategy involving informed guessing and backtracking. Moving beyond imitation learning, we seek to minimize the number of guesses until reaching a solution. This is achieved using depth-1 guessing, showing empirically that almost all Sudoku can be solved using the puzzle's rules with at most one guess. We provide a rigorous analysis of this setup formalizing its connection to a contextual variant of $\textit{Min-Sum Set Cover}$, a well-studied problem in algorithms and stochastic optimization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies transformer models for combinatorial problems, with a specific focus on NP problems. This has been studied a lot in recent years by the community, as covered in the related works section of the paper. The authors obtain gains of >10% for full solution accuracy using architectures from prior work with their new data generating library, which also allows for test set generation. \n\nThe authors then make three key addtions to obtain another gain of >6% to obtain an accuracy of 99%. Firstly, a method for representing the solving process as a series of discrete actions, secondly allowing the model to learn backtracking through curated training data and finally using a novel loss function which is not one hot to allow for probability of the model to be placed on all correct possible next moves. The change is loss function is backed by both theoretical and empirical results. One empirical validation of this new loss function is in Figure 6 where the new loss function allows more than 80% of puzzles to be solved in fewer guesses than an oracle.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n- Strong related works section which places the paper well in context.\n- The introduction and study of a novel loss function for combinatorial problems and new method to train models to learn to backtrack.\n- The promise to open source a new library to make it easier for the community to study NP problems in the future.\n\n# Weaknesses\n\n- There is a lack of discussion on the additional time/memory taken to train and inference using the new method. The authors compare to non-backtracking techniques which should use less tokens/compute to generate their solutions, although these come with the down sides the authors highlight throughout their paper there are other ways to compare to these methods on a FLOPs/output-tokens comparable axis.\n\t- For example, if the new backtracking technique takes twice the compute/tokens then we could run the non-backtracking approaches twice with a non-zero temperature to see if they generate the correct solution either time. One thing to note is that as the multiple attempts can be completed in the batch dimension the exact comparison here is intricate but inference FLOPs or total tokens output could be good options to make comparisons fair. \n\t- Additionally, in Table 1 the authors analyse at reasoning models for Sudoku. A reasonable baseline could be do GRPO directly on sudoku problems, as I believe “Aha movement” in GRPO (https://arxiv.org/pdf/2501.12948) does allow for backtracking.\n\n- It would be nice to contextualise in the conclusions outside of the language modelling community. For example, in the context of how fast traditional SAT solvers are, and highlight the future directions this work allows, highlighting the connection to all NP-Complete problems more directly for non topic specific readers.\n\nQUESTIONS:\n1. Is the Deepseek-R1 “Time” in number in table 1 large as it was run locally and all others are run on APIs? Either way, it would be very useful to have exact API references for all models in Table 1.\n\n2. Where would the r,c,v + multiple targets line be on Figure 3 left?\n\n3. Please see weakness 1, on \"additional time/memory taken to train and inference using the new method.\"\n\nDiscussion/progress on weakness 1 is most likely to impact my score.\n\nLIMITATIONS:\nI would like to see a little more on the limitations of this work compared to specialised/classical SAT solvers.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework for solving combinatorial problems such as Sudoku using a standard decoder-only Transformer model. The key innovation lies in combining supervised imitation learning of basic Sudoku rules with a trial-and-error Depth-First Search (DFS) strategy. This enables the model to make informed guesses, backtrack when encountering conflicts, and eventually solve problems that cannot be addressed by heuristic rule application alone. Additionally, the paper introduces a novel loss function inspired by the Min-Sum Set Cover problem to optimize inference efficiency. The proposed method achieves near-perfect accuracy (99%) on Sudoku and 1-in-3 SAT tasks, outperforming prior neural and neuro-symbolic baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n---\n- The model leverages a pure Transformer architecture (GPT-2 variant) without requiring architectural modifications or external solvers.\n- The model achieves 99% accuracy on standard Sudoku, demonstrating strong reasoning ability.\n- The authors provide a custom C-based puzzle generator and training transcript generator to support large-scale evaluation and reproducibility.\n- The authors demonstrate fair comparisons across multiple datasets and training regimes (Kaggle, RRN, Random) in their experiments.\n\nWeaknesses\n---\n- **Length and Structure of Introduction**: The introduction is overly long and merges motivation, background, and methodology. This impairs clarity about the specific contributions. Reorganising the content (e.g., by moving methodology details to a separate section) would improve readability.\n- **Lack of Training/Inference Time Reporting**: While the motivation includes time-per-puzzle as a crucial factor (Table 1), the paper does not report actual training or inference times, which weakens claims about efficiency.\n- **Insufficient Detail on Generalisation to NP Problems**: While the method is said to be applicable to general NP tasks, concrete explanations on how domain-specific design choices (e.g., tokenisation 111--999, rule application mechanics, backdoors) translate to other problems are not enough. The treatment of the 1-in-3 SAT task is not found in the main text.\n- **Neural Network Specification**: The paper does not clearly explain what inputs the Transformer receives and what outputs it produces. For example, while the transcription format and guess strategy are described, the Transformer input/output format and parameterisation are not fully formalised.\n- **Theoretical Contributions Need Clarification**: In Section 3, the comparison between two loss functions (L1 and L2) is said to be grounded in theory, but the intuitive significance of the findings is underexplained. More discussion of what the results mean practically would help.\n- **LLM Framing and Generalisation**: It remains unclear how this model fits into the LLM paradigm. Unlike general-purpose LLMs that process natural language prompts, this model uses highly structured tokens. The authors should clarify whether the model is applicable to natural language-stated logic puzzles or if its scope is limited to symbolic encodings.\n- **Missing Related Work on SAT Solvers**: The related work focuses heavily on Sudoku solvers. More references with neural SAT solvers (e.g., NeuroSAT, GNN-based approaches) would help understand the work within the broader literature.\n\nQUESTIONS:\n- What exactly is supervised imitation learning in your setup? A concise explanation would be helpful for readers unfamiliar with this technique.\n- What optimiser, GPU type, and training time (in GPU hours) were used? Reproducibility would benefit from detailed training settings.\n- The algorithm starting from the line 214 seems to be the main contribution. While the generation of training transcripts is explained, how exactly is this used during training and inference? What is the architecture of the network that takes these transcripts and learns from them?\n- How does the algorithm described around line 214 differ from that of the Causal Transformer? Based on your explanation, the main differences seem to be: (1) using 4 inference rules instead of 7, (2) using a more compact encoding by removing commas between row, column, and value triplets, and (3) adopting a multiple next-token prediction loss. Are there any other differences? Given that the Causal Transformer is a key baseline, the paper should explain this comparison more clearly.\n- In Table 2, why is there no test result for Causal Transformer on the Random dataset? Is it infeasible to evaluate?\n- In Table 2, is the Causal Transformer trained on the Kaggle filtered dataset only? It would be important to determine whether the performance gain of your model is due to the dataset change during training or the proposed methods in Section 2. An ablation study is needed to identify which factors contributed most to the performance improvement.\n- What do the terms “lower bound” and “upper bound” in Figure 6 represent? Is the lower bound equivalent to the “ours” model described in Section 2? The upper bound seems to assume access to the correct values of the remaining cells, functioning as an oracle. In that case, what does “guess” mean when the correct values are already known? Does it refer to steps where none of the four Sudoku rules can definitively determine the next number?\n- The paper’s approach to forward guessing and backtracking in Sudoku closely resembles the DPLL algorithm used in SAT solving. In DPLL, deterministic rules are applied until no further progress is possible, then a guess is made; if a contradiction arises, the process backtracks. A notable extension is CDCL (Conflict-Driven Clause Learning), which learns from conflicts to avoid similar mistakes. Adopting such strategies could help extend this method to more general NP-class problems.\n\nLIMITATIONS:\nThis paper presents a framework that combines symbolic reasoning with Transformer-based learning in a novel way. Its success on Sudoku shows promise. However, the paper would benefit greatly from improved clarity in technical exposition, more extensive discussion of generalisability to broader NP tasks and general-purpose natural language processing, and inclusion of relevant related work in SAT-solving and symbolic learning in LLMs. If these areas are addressed, this work is likely to make a meaningful contribution to the field of combinatorial reasoning with language models.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA new strategy to tackle NP problems using transformers is proposed (exemplified by the classic Sudoku puzzle) - without external tools or code. The training and operation of the model is in 2 phases: learning basic Sudoku rules via imitation learning, and learning to guess cell values efficiently via a depth-first search - eventually getting to single-guess solution flows. A generator for Sudoku puzzles in a uniform distribution - for training and testing - is developed and published.\n\nThe new method is applied to several test datasets - achieving SOTA results.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nA solid work - in terms of engineering, experimental design, simplicity of approach, and efficiency.\n\nWell written and clearly explained (even if a few concepts in the intro become clear only later on), with high quality figures 2 and 5.\nSOTA on Sudoku benchmarks. \n\nPotentially applicable to a wide range of puzzles and mathematical challenges - even broader than the authors mention explicitly.\n\nWeaknesses:\n\nThe way I understood the system, there are 2 main parts: teaching the transformer the 4 basic Sudoku rules - imitating a deterministic algorithm - and teaching to make good guesses, specifically such that a single guess will bring to a full solution. While the second part can naturally leverage the flexible nature of networks, the first part seems more like a test of the tokenization and dataset generation choices - not a use case that would naturally benefit from DL.\n\nThe authors apply their method to other NP satisfiability problems to show generalization - specifically to 1-in-3 SAT, but it is only stated in the main paper - no details or description given. The information is in Appendix G (incorrectly called H in line 176).\n\nWhile LLMs are mentioned as motivation and potential impact of the paper (both in the intro and summary), this system resembles more an RL setting, or maybe action/time series data. I believe this approach is useful even without connection to LLMs (there are many transformer systems that have nothing to do with natural language) - but the claim should either be strengthened or removed.\n\n---\n\nLine 155: *in\n\nLine 310: “that the L1 is theoretically” - “the” should be removed.\n\nQUESTIONS:\nLine 141: The oracle is the same system that generated solutions for the imitation training stage? If so, I’m not sure I understand how we know that its solutions are optimal - or how close to optimal they are? It’s still a strong result if the transformer outperforms the training data substantially - just not clear how to attribute it specifically to the “backdoor guess” step.\n\nFigure 3 - if the r,c,v representation counts each step as 3 tokens, so the number of actual steps seen by the model is 3 times lower - yet achieves similar (even if slightly lower) results? I probably misunderstood something.\n\nLine 293: Is the “backdoor property” a known feature of Sudoku? Is it an empirical observation based on your generation of training data?\n\nHave you tried experimenting with a hybrid approach: apply the rules deterministically + train the model to guess only once the simple logic code gets stuck? Presumably it would lower the required training set, perhaps significantly, and leverage the strength of each part.\nThis can better separate the 2 functionalities that the model learns simultaneously.\n\nHave you tried rephrasing the training setting (and loss) in terms of standard RL metrics? The “moves” of the game are the guess steps, the reward (win/lose) is simply whether the board reaches an “e” or a “d”.\nThis is not mandatory of course - just seems to me like a more natural setting than learning transcripts of solutions.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework to teach Transformers to solve combinatorial problems, using Sudoku as the primary testbed. The method combines imitation learning (of simple rule-based logic) with search, using Depth First Search informed guessing and backtracking. The authors introduce action-level tokenization and multi-target training strategy, achieving 99% accuracy on Sudoku and generalizing to 1-in-3 SAT which is an NP hard problem. The second contribution is a theoretical and empirical analysis of minimizing solution length by formulating the guessing optimization as a contextual Min-Sum Set Cover problem, resulting in a new loss function that outperforms baseline cross-entropy loss.\n\nSTRENGTHS AND WEAKNESSES:\n1) The proposed method is well designed, clearly described, and simple.\n2) This paper explores the use of LLMs for combinatorial reasoning, showing that trial-and-error with backtracking can be learned effectively.\n3) The reframing of guessing optimization as a contextual Min-Sum Set Cover problem (and the new loss function as a result) is interesting.\n\nWeakness:\n1) The DFS-based backtracking search may not scale to harder combinatorial problems or larger sizes without further optimization. Authors didn't discuss the possible solutions to address problem size.\n\n2) Evaluation is only limited to Sudoku and 1-in-3 SAT. The paper claims the effectiveness of using the method on general NP problems, but fails to provide more evaluations to support this claim. \n\n3) This paper has missed discussing and comparing important related work such as: \"Neural Combinatorial Optimization with Reinforcement Learning\" Bello et al., 2016 and the many follow up works that came out afterwards.\n\nOverall, The DFS-based backtracking search is an effective way to solve general NP problems, however, the scalability of the method on larger problems is challenging. This paper presented a way to integrate imitation learning of simple rule-based logic with Depth First Search (DFS) informed guessing and backtracking. The proposed optimization framework is limited to depth-1 in DFS and non-adaptive guessing. This will limit the application of this work on general NP problems. Moreover, the paper claimed to be effective to solve general NP problems, but failed to provide diverse evaluation to support this claim.\n\nQUESTIONS:\nHow does the proposed framework perform on Sudoku-Bench?\n\nThe DFS-based backtracking search is hard to scale. How does the model perform on larger Sudoku grids or harder SAT variants? Are there known scalability bottlenecks? Is there any further optimization needed? \n\nHave the authors studied the impact of allowing deeper DFS (beyond depth-1)? What is the trade-off in performance and runtime?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}