Title: Memory-Enhanced Neural Solvers for Routing Problems
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

combinatorial optimisation

routing

memory
Abstract: Routing Problems are central to many real-world applications, yet remain challenging due to their (NP-)hard nature. Amongst existing approaches, heuristics often offer the best trade-off between quality and scalability, making them suitable for industrial use. While Reinforcement Learning (RL) offers a flexible framework for designing heuristics, its adoption over handcrafted heuristics remains incomplete. Existing learned methods still lack the ability to adapt to specific instances and fully leverage the available computational budget. Current best methods either rely on a collection of pre-trained policies, or on RL fine-tuning; hence failing to fully utilize newly available information within the constraints of the budget. In response, we present MEMENTO, an approach that leverages memory to improve the search of neural solvers at inference. MEMENTO updates the action distribution dynamically based on the outcome of previous decisions. We validate its effectiveness on Traveling Salesman and Capacitated Vehicle Routing problems, demonstrating its superiority over tree-search and policy-gradient fine-tuning; and showing that it can be zero-shot combined with diversity-based solvers. We successfully train all RL auto-regressive solvers on large instances, and verify MEMENTO's scalability and data-efficiency: pushing the state-of-the-art on 11 out of 12 evaluated tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new approach to solve routing problems using learning based methods. Existing methods include constructive heuristics, that learn a policy to construct a solution but do not leverage the insight from earlier tried solution on the same problem instance. Other existing methods do this, but with additional limitations such as being applicable only to heatmap-based policies or only learning embeddings. The proposed solution is architecture-agnostic: it learns to modify the logits generated by a 'base policy' based on memory of choices made at the same node on the same instance. The instances in this memory are combined using a learned 'rule'. Experiments show this method capable of improving the performance of the base method to the state of the art, even on large instances (500 nodes) and even if combined with a different base method at inference time.\n\nSTRENGTHS AND WEAKNESSES:\nQuality - Strenghts\n- The main claims in the paper are supported empirically. Experiments are well set up, thorough, on relevant and subfield-appropriate environments, with a suitable choice of and number baselines. (although only two different problems, TSP and CVRP, at different sizes, are used.) Relevant analysis and ablations are shown, such as the comparison to REINFORCE, the zero-shot combination, ablations on policy inputs.\n\nQuality - Weaknesses\n- Formulation: The formulation in 3.1 does not seem to accomodate the policy being conditioned on earlier samples, which does not seem to fit the approach proposed later. \n- The comparison to REINFORCE could perhaps be stated a bit more carefully. I understand the comparison of two processes that based on data would change the logit of future actions. However, it should also be stated that these were defined for very different purposes: long-term gradual improvement of a policy versus immediate (and somewhat greedy) selection of an action. While the comparsion this nuance could be better transmitted. \n-The paper states that existing work on leveraging prior samples have additional limitations such as being applicable only to heatmap-based policies or only learning embeddings. One exception that could be mentioned is \"training graph neural networks with policy gradients to perform tree search\" by Macfarlane at all (Neurips Deep RL workshop 2022). \n\nClarity - Strengths\n- The structure of the paper is generally clear, the use of language is generally clear and appropriate, figures are used effectively to illustrate the methods and results. \n\nClarity - Weaknesses\n- Abstract could make more clear that approach leverages earlier attempts on the same instance; also to readers who wouldn't immediately understand what 'budget' refers to in this setting. \n- Some discussion and presentation of the results is misplaced. The structure could be improved by moving Fig. 1 to Section 4 (now on the first page!) and to remove the forward reference to Fig.4 from Section 2. The metrics in Fig 1 (left) should be better explained. \n\nSignificance\nThe proposed method achieves a consistent performance across almost all conditions and environments investigated. The effect size is sometimes modest, which makes sense on these environments where performance is slowly satisfying. It was impressive to see good performance from the POMO family of auto-regressive methods on size-500 instances for the first time. Experiments are limited to TSP and CVRP so it is not known how significant the proposed method is on other routing problems or even non-routing combinatorial problems. Overall, I would consider the method to be a quite significant contribution to this subfield. \n\nOriginality\nThe main idea of the paper is somewhat similar to those of existing work, but getting this to work at state of the art level required several original insights, design choices, and modifications. Overall I'd rate the originality as good. \n\nMinor comments\n- line 157 mentiond only (ii, iii, iv) are sufficient for the reinforce update, but this should include the state as well.\n- sentence in line 293 lacks a subject\n- it is helpful to include the metric shown in the figure caption or legend. This is missing from figures 4, 6, for example (it is explained in the text, but that makes it a bit harder to find)\n\nQUESTIONS:\n- As representation of the state, your method only uses the current node visited. Have you experimented with other representations of the state (such as an (embedding of) the sequence so far), and why do you think using only current node gives the best result? \n\n- Why do you think it helps the method to have as input the remaining budget when action in memory was taken? It seems to me that to maximize the reward function (improvement over best reward so far) this should not have impact on the optimal policy. Perhaps it is a proxy for \"best reward so far\" (which should help the policy)? Did you consider \"best reward so far\" as feature for the memory module?\n\n- How well do you expect this method to generalize to other routing problems and to non-routing optimization problems?What is the motivation for only considering TSP and CVRP?\n\nLIMITATIONS:\nThe checklist mentions error bars are reported, however, none of the tables give error bounds (Fig. 5 does, though).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MEMENTO, a memory-based method for online fine-tuning of neural combinatorial optimization solvers. MEMENTO enhances a pretrained constructive policy by incorporating a learned residual policy that adjusts action probabilities during inference. A memory module is used to store features from past solution trajectories—such as actions taken, associated rewards, and contextual signals—which are processed by an MLP to guide future decisions. An improvement-based reward signal enables MEMENTO to adapt the policy on-the-fly to each problem instance. Experiments on TSP and CVRP instances with up to 500 nodes demonstrate that MEMENTO consistently outperforms strong baselines, achieving effective instance-specific adaptation within constrained computational budgets.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1) The paper is well-written, with clear exposition and strong organization. \n\n2) A notable contribution of this work is its focus on incorporating computational budget constraints into the design of a construction-based neural CO solver.\n\n\n## Weaknesses\n\n1) Despite the considerable additional effort—such as introducing a memory module, fine-tuning the base model while training the auxiliary MLP concurrently—the performance gain of MEMENTO over the much simpler baseline, EAS, is not particularly compelling.\n\n2) To better highlight what I believe is one of the core strengths of the method—the incorporation of budget constraints—the authors should evaluate performance across a wide range of computational budgets: low, moderate, and high. If the meta-learning approach of the authors are working as claimed, the results will show that the model trained for the large budget setting performs worse than the low-budget setting model in the short run, but eventually surpasses it as more computation is used.\n\n3) The improved update rule illustrated in Figure 4 is intriguing, but the methodology behind generating the figure is not well explained. To plot such a graph, one would need to fix all other input variables to the MLP. Was the figure created by averaging over all nodes? All trajectories? Clarifying this process would help readers better understand the nature of the learned update behavior.\n\n4) Furthermore, if the authors can extract and interpret the underlying logic of the MLP that leads to improved update rules over REINFORCE—as Figure 4 suggests—this direction deserves deeper exploration. For example, does the MLP promote stronger exploration early in the process and gradually shift toward exploitation? Does it induce more aggressive updates at the beginning or closer to the end of the budget? Addressing such questions would not only help validate the effectiveness of the method but also provide valuable insights for the broader research community.\n\nQUESTIONS:\nnone\n\nLIMITATIONS:\nnone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MEMENTO (Memory-Enhanced Neural Solvers), a novel approach designed to enhance the search capabilities of neural solvers for Routing Problems, particularly at inference time. The core motivation is to improve the adaptability of learned heuristics to specific problem instances and better utilize available computational budgets, an area where existing Reinforcement Learning (RL) based methods often fall short compared to handcrafted heuristics.\n\nMEMENTO achieves this by integrating a memory mechanism into the neural solver's inference process. This memory dynamically updates the action distribution based on the outcomes of previous decisions, allowing the solver to learn from its own search history within a single instance. \n\nThe key contributions of this work are:\n\nA novel memory-enhanced framework: MEMENTO leverages a dynamic memory to guide the search process of neural solvers at inference time.\n\nDynamic action distribution update: The memory mechanism allows the model to dynamically update its action distribution based on previously visited nodes or partial solutions, enabling better adaptation to the current instance.\n\nSuperior performance and adaptability: MEMENTO is validated on Traveling Salesman Problem (TSP) and Capacitated Vehicle Routing Problem (CVRP), demonstrating superior performance compared to traditional tree-search methods and policy-gradient fine-tuning.\n\nZero-shot combination with diversity-based solvers: The paper shows that MEMENTO can be effectively combined with diversity-based solvers, further improving solution quality.\n\nScalability to large instances: The approach successfully trains RL autoregressive solvers on large instances, indicating its potential for real-world applications.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nQuality: MEMENTO presents a well-conceived and technically sound approach to improve neural solvers. The integration of a memory mechanism at inference time is innovative and addresses a critical limitation of existing RL-based heuristics (lack of instance-specific adaptation). The experimental setup is robust, comparing MEMENTO against strong baselines including tree search (POMO with sampling), policy gradient fine-tuning, and combining it with diversity-based methods. The results consistently demonstrate MEMENTO's superiority across TSP and CVRP benchmarks, including larger instances, which strongly supports the claims.\n\nClarity: The paper is well-written and easy to follow. The problem statement regarding the limitations of current RL heuristics is clearly articulated. The MEMENTO framework, including the motivation for memory, the memory update mechanism, and its integration with the neural solver, is explained with excellent clarity. Figures (e.g., Figure 1 illustrating the framework) are highly effective and aid comprehension. The mathematical formulations are precise and enhance the understanding of the memory dynamics.\n\nSignificance: MEMENTO offers a significant contribution to the field of neural combinatorial optimization. By enabling instance-specific adaptation at inference time without requiring expensive retraining or complex tree-search algorithms, it bridges a crucial gap between learned heuristics and traditional handcrafted heuristics in terms of flexibility and budget utilization. This improvement in adaptability and quality, especially for larger instances, has immense practical implications for real-world routing problems. The ability to zero-shot combine it with diversity-based solvers further enhances its practical utility.\n\nOriginality: The core idea of \"memory-enhanced\" inference for neural combinatorial optimization is highly original. While memory networks exist in other domains, their specific application here to dynamically update action distributions in an autoregressive routing solver at inference time, based on an evolving partial solution, is novel. The method for deciding what to store and how to retrieve from memory is also an original design choice that leads to strong empirical results.\n\nWeaknesses:\n\nMemory Content and Retrieval Strategy: While the paper justifies the chosen retrieval strategy (retrieving from the same node), a more detailed exploration of alternative memory content (e.g., more complex partial solution representations beyond just the current node's embedding) and retrieval strategies (e.g., k-nearest neighbors based on solution embeddings, as briefly discussed but dismissed for cost reasons) could provide deeper insights. The current strategy might be optimal for the tested problems, but its generalizability to problems with richer state information is less clear.\n\nComputational Overhead of Memory: While the paper states that the simplified retrieval approach improves scaling compared to k-nearest neighbors, a more explicit quantitative analysis of the computational overhead introduced by the memory operations (storage, retrieval, update) as a function of problem size (N) and number of attempts (K) would be beneficial.\n\nLimitations of \"Zero-Shot\" Combination with Diversity: While MEMENTO can be combined zero-shot with diversity-based solvers, the paper could briefly discuss potential limitations or scenarios where this combination might not be as effective, or where a more integrated design might be preferable.\n\nQUESTIONS:\nQuestion: The paper demonstrates that retrieving from the same node is an effective proxy for similarity. Have the authors explored more complex memory contents (beyond just the current node's representation) or retrieval strategies in detail, perhaps even briefly experimenting with them for smaller instances to understand their theoretical potential before dismissing due to computational cost? For example, storing more holistic partial solution embeddings or using different similarity metrics.\nSuggestion: Provide a brief qualitative or quantitative discussion (perhaps in the appendix) on the trade-offs involved in different memory content and retrieval strategies. While the chosen method is practical, demonstrating why others are less suitable (beyond just computational cost) would strengthen the design choices.\n\nQuestion: The paper mentions that the simplified retrieval approach was chosen for better scaling. Could the authors provide a more explicit quantitative analysis of the computational overhead introduced by the memory update, storage, and retrieval operations within MEMENTO? How does this overhead scale with the number of nodes (N) and the number of search attempts (K)?\nSuggestion: Include a table or graph in the appendix showing the breakdown of inference time, distinguishing between the core neural solver's time and the overhead incurred by MEMENTO's memory operations, across different problem sizes and K values. This would offer a clearer picture of MEMENTO's practical efficiency.\nCriterion for Score Change: A detailed computational analysis of the memory operations would improve the Quality of the experimental evaluation and enhance the paper's practical Significance.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
RL agent to solve specific routing problems which also utilizes history of past decisions and their outcomes for crafting better updates.

STRENGTHS AND WEAKNESSES:
Strengths: 

1. I like the overall motivation of the approach to give history of previous time steps to derive a better 'update steps' for the CO solver. 

2. Extensive experimental evaluation shows promise. 

Weaknesses:

3. I would have liked more details about the method itself in the main paper. The content between Lines 129 - 195 is too abstract to get the exact details of the approach. I know about more details in the Appendix but let us not underestimate the importance of sharing 'research thoughts/ideas/explanation' and overemphasize the importance of experiments. Even the teaser Figure in the first page is not so important.   

4. Similar to above Figure 3 does not add a lot of value. I would rather prefer exact details in text. Similarly Lines 133-135: it can be combined with ...., should rather be elsewhere. I would like to know what is being done in this paper! I can keep going to give further examples but I think the above ones are enough to get my point across.  PS: I like Algorithm 1 in the Appendix, perhaps it can be put in the main paper and using that as means of describing the method?

QUESTIONS:
5. Do you see the proposed approach also being useful for other cases of CO (non-routing) problems?
6. The word node occurs multiple times in the paper but it is not exactly defined anywhere. Does by node you mean a decision variable of the CO problem?
7. How do base policy module and memory module interact to produce the resulting policy? (Seeing Figure 2).

LIMITATIONS:
8. Exposition of the paper needs to be improved.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01