Title: CORectifier: Hierarchical Trajectory Rectifications Boost Reinforcement Learning for Combinatorial Optimization
Primary area: learning on graphs and other geometries & topologies
Keywords: Neural Combinatorial Optimization, Reinforcement Learning, Imitation Learning, Hierarchical Rectification

Abstract: In the rapidly advancing field of Neural Combinatorial Optimization (NCO), Reinforcement Learning (RL) stands out for its ability to naturally enforce complex constraints. However, mainstream RL-based methods suffer from reward sparsity and sample inefficiency in the vast combinatorial action space, leading to ineffective training, subpar performance, and poor scalability. To address these challenges, we propose , a novel NCO solver learned with hierarchical supervision incorporated to regularize the arbitrary RL exploration. Technically, we design a "rectification" mechanism for training: partial policy-predicted actions in a trajectory are probabilistically replaced with high-quality segments from reference solutions. Distinct from heatmap-guided Supervised Learning (SL) or vanilla Imitation Learning (IL) methods, CORectifier prompts the model with tri-level optimal signals, operating at the batch, instance, and sub-instance levels with fragments of random lengths injected at random decision steps. This Rectified RL (RRL) paradigm helps develop optimality-aware and sample-efficient RL learners while maintaining their sequential-decision manner for constraint satisfaction, delivering a new perspective to hybridize RL and SL/IL for solving CO problems with improved flexibility and utility of limited supervisory data. Empirical results on TSP, ATSP, and PCTSP validate the superior training effectiveness of CORectifier. When solving with a consistent greedy policy, CORectifier achieves up to 59.7% and 19.3% performance improvements over compared RL-based and SL-based methods (all without any augmentations or multi-sampling), respectively, and reduces the performance gap on TSP-500 (sufficiently large for RL solvers) by up to 89.01%.