VColRL: Learn to solve the Vertex Coloring Problem using Reinforcement Learning
Authors: Abhinav Anand
Subrahmanya Swamy Peruru
Amitangshu Pal
Venue: Accepted by TMLR ()

Abstract: The Vertex Coloring Problem (VCP) is a fundamental NP-hard problem with applications in wireless networks, compiler design, scheduling, etc. We present VColRL, a deep reinforcement learning (DRL) framework that learns to color graphs quickly by leveraging a reduction-based approach that progressively reduces the graph at each step. The core novelty of VColRL is a new Markov Decision Process (MDP) formulation tailored for VCP that assigns colors to multiple vertices at each step, incorporates a rollback mechanism to revert all conflicting vertices to the undecided state, and employs a reward function designed to minimize the highest-indexed color used. Experiments on synthetic and benchmark graphs show that VColRL improves color usage over optimization solvers and prior learning-based methods, remains competitive with search-based heuristics and metaheuristics, and achieves fast runtime, while generalizing well to diverse graph families despite being trained only on synthetic graphs from a single family.

Published in Transactions on Machine Learning Research (09/2025)
VCoIRL: Learn to solve the Vertex Coloring Problem using Reinforcement Learning
Abhinav Anand Deportment of Computer Science and Engineering II Kanpur
abhimavananddcee.ittk..
Subrahmanya Swamy Peruru
swammyyPhtkkkccomm
Deportment Electrical Engineering IIT Kampur
Amitangshu Pal
amitannghauucsee.ik..ooo
Department of Computer Science and Engineering IIT Kanpur
Reviewed on OpenReview: /ttps://opense www.r/veeeee.e.e...mm net/ forum? id- a9AQR
Abstract
The Vertex Coloring Problem (VCP) is a fundamental NP-hard problem with applications in wireless networks, compiler design, scheduling, etc. We present VCoIRL, a deep rein- forcement learning (DRL) framework that learns to color graphs quickly by leveraging a reduction-based approach that progressively reduces the graph at each step. The core nov- elty of VCoIRL new Markov Decision Process (MDP) formulation tailored for VCP that assigns colors to multiple vertices at each step, incorporates rollback mechanism to revert all conflicting vertices to the undecided state, and employs reward function designed to minimize the highest-indexed color used. Experiments on synthetic and benchmark graphs show that VColRL improves color usage over optimization solvers and prior learning- sermingg-baedd methods, remains competitive with search-based heuristics and metaheuristics, and achieves fast runtime, while generalizing well to diverse graph families despite being trained only on syuthetic graphs from a single family.
1
Introduction
The Vertex Coloring Problem (VCP) assigns colors the vertices of graph using the minimum number of colors, such that no two adjacent vertices share the same color. The optimization version of the VCP is NP-hard problem, whereas the decision version is NP-Complete (Garey et al. 1974 Garey & Johnson,
1990). VCP is used to model diverse real-world applications like register allocation and spilling in compilers (Chaitin 1982), frequency assignment in wireless networks, ezam limetabling, sports scheduling, aircraft scheduling, physical layout segmentation, map coloring, etc (Ahmedl 2012). There are several fields, such as wireless communication and compiler design, where obtaining solutions quickly is critical dynamic and rapidly changing requirements. In wireless networks, the channel allocation task is modeled VCP, where transmitters within interference range must be assigned different frequencies to avoid signal collisions (Hale) 1980). In dynamic wireless communication environments such as Mobile and Vehicular Ad hoc Networks (Hocbeke et al.) 2004 Al-Sultan et al. 2014), rapid reallocation of the resources is neoded to adapt to changing lopologies. Similarly, in compilers, the register allocation task is modeled as VCP. where variables with overlapping lifetimes must be assigned distinct hardware registers (Chaitin et al. 1981). Finding quick solution is crucial just-in-time (JIT) compilation systems like Numba (Lam et al. 2015). to ensure fast execution of the programs. VCP is a classic example of a combinatorial optimization, subfield of mathematical optimization concerned with selecting the best solution from finite, often vast.
1976
meet

---

Published in Transactions on Machine Learning Research (09/2025)
set of discrete possibilities. Other popular combinatorial optimization problems are the Traveling Salesman Problem (TSP) (Vigi 1831), the Maximum Independent Set. Problem (MIS) (Miller & Muller 1960 the Maximum Clique Problem (Bomze et al. 1999), the Minimum Vertex Cover Problem (Dinur & Safra 2005 ctc.
Various approaches to solve combinatorial optimization problems: Unfortunately, many combina- torial optimization problems are NP-hard (Papadimitriou & Steiglitz, 1998), and therefore, finding exact solutions becomes computationally infeasible for large instances. To this end, rescarchers have developed heuristics, which are fast and problem-specific,
well as metaheuristics, which provide more general strate- gies to find high-quality solutions for these problems. For instance, FastColor (Lin ct al. 2017 is hcuristic, whereas TabuCol (Hertz& & Werra 1987), based on tabu search, is a metaheuristic that often provides good solutions for the VCP. For problems that can be modeled as integer linear programming (ILP) Graver 1975)
or mixed-integer linear programming (MILP) (Vielma 2015), ; advanced solvers such as Gurobi Optimization
Studio (Gurbi Optimization 2024) and IBM ILOG CPLEX Optimization Studio (IBM) 2024) can compute
exact solutions. These solvers employ sophisticated techniques, including branch-and-bound (Morrison el al., 2016) and cutting-plas methods (Goffin & Vial 2002 Lee et al. 2015), to handle large-scale instances with high precision. Despite their powerful capabilities. the time required for these solvers to find exact solutions increases substantially as the problem size grows Luppold et al. 2018).
Beyond heuristics, meta-heuristic and optimization solvers, data-driven methods offer an alternative for tackling complex optimization problems, reducing reliance on extensive domain expertise and rule-based algorithms. Rather than relying on fixed sets of rules, these methods leverage machine learning (ML) (Jordan & Mitcchell, 2015) to learn directly from data. Over the past decade, the rise of neural networks (Bishop 1994) and advanced ML techniques has further strengthened these approaches, making them powerful and versatile. While supervised learning (Cunningham et al. 2008) offers one path under the data-driven paradigm, its reliance on labeled datasets poses significant challenges, as obtaining exact solutions for hard optimization problems is often difficult. Consequently, alternative methods are sought that do not roly on labeled data. Reinforcement learning (RL) (Wiering & Van Otterlo, 2012) offers a promising direction, as it does not rely on labeled data but instead frames the problem as a Markov Decision Process (MDP) (Puterman] 1990). enabling agents (models) to learn optimal strategics through iterative interactions with the environment and the feedback received (Sutton & Barto 2018 Ernst & Louette 2024). However, traditional RL methods struggle to scale effectively in environments with large state or action spaces. To address this limitation, researchers introduced deep reinforcement learning (DRL) (Arulkumaran et al. 2017), which integrates RL with neural networks to handle high-dimensional spaces in order to model complex environments and discover intricate patterns that are difficult. to capt ure with traditional RL methods (Lapan) 2020). RI- basod approaches are proven useful in areas such as robotics (Morales et al., 2021), autonomous systems (Kiran et al., 2021), game playing (Dong et al.] 2020),, etc. To solve graph-related problems with data-driven methods, graph neural networks (GNNs) (Scarselli et al.) 2008) are employed as function approximators, as they effectively capture structural dependencies among vertices through message passing. This ability is particularly valuable for problems such as the VCP, where deep understanding of graph topology is essential for achieving optimal color assignments. GNNs in DRL yield a powerful non-convex optimization framework that exploits graph structure and topology without labeled data.
Our contributions: To this end, we introduce a GNN-basod novel deep reinforcement learning (DRL)
framework to address the VCP. Our key contributions are as follows:
We design novel MDP formulation for the VCP that colors multiple vertices at each step, employs a hard rollback mechanism to resolve conflicts by reverting all conflicting vertices, and introduces a max-color reward to reduce label ambignity, encourage compact color assignments, and stabilize training. The MDP is derived by exploring different rollback, action, and reward strategies to determine the configuration most effective for the VCP.
Building on this MDP formulation, we develop VCoIRL, a GaNd-based DRL framework that em- ploys Proximal Policy Optimization (PPO) (Ahn al. 2020) and leverages the GraphSAGE archi- tecture (Hamilton et al. 2017) to improve runtime efficiency.
2

---

Published in Transactions on Machine Learning Research (09/2025)
We conduct extensive empirical evaluations synthetic and benchmark graphs, showing that VColRI. improves color usage over optimization solvers and prior learning-based methods, remains competitive with search-based heuristics and metaheuristics, and achieves fast runtime while gener- alizing well to diverse graph familics despite being trained only on one type of synthetic graph. We also release the VColRL code for community use.
Paper organization:
The rest of the paper is organized follows. Section 2 reviews the related work. Section 3 presents the framework of VCoIRL, including details on the MDP formulation, model training, identifies the best MDP configuration
hyperparameters, and the handling of incomplete solutions. Section 4 and evaluates the performance of VColRT, comparing it against several baselines across diverse families of benchmark and synthetic graphs. The generalization capability and scalability of VCoIRL are also discussed in this section. Finally, Section 3 concludes the paper and outlines future directions.
2 Related Work
In the realm of optimizations, the Vertex Coloring Problem (VCP) is a well-known NP-hard combinatorial problem (Garey et al. 1974 Garey & Johnson 1976 1990). Solutions to this problem can be categorized into two main approaches: conventional and based.
Conventional approaches: This category includes heuristics such as DSATUR (San Segundo 2012 Welsh-Powell (Welsh & Powell| 1967), Fast (Lin ct al. 2017), as well as metahouristic approaches such as Simulated Annealing (Bertsimas & Tsitsiklis 1993), Tabu Search (Glover 1989), etc. These algorithms are capable of generating high-quality, near-optimal solutions. Furthermore, there are mathematical optimization solvers like Gurobi (Gurobi Optimization 2024 and CPLEX (IBM 2024), which provide exact solutions but suffer from rapidly increasing execution time as the problem size grows.
Machine learning approaches: Several studies have used supervised learning for solving the VCP. Das et al. (019 have introduced a supervised learning approach where a deep learning model, using Long Short- Term Memory layers followed by a correction phase, is trained. Over time, researchers have recognized the need for graph-specific architectures to better address the complexities inherent in vertex coloring problems. Lemos et al. (2019) has employed graph neural network (GNN) (Scarselli et al., 2008) to predict the chromatic number of graphs, leveraging GNNs' ability to capture graph structures. Similarly, [[aaz et al. (2022) has utilized GNNs to solve the VCP, focusing on efficiently finding the chromatic number of large graphs.
However, as graph sizes increase, oblaining ground truth using conventional approaches becomes impractical, leading to the emergence of reinforcement learning as a promising solution. [Huang al. (2019) has adapted AlphaGo Zero (iilver et al. 2017] with graph embeddings by introducing a novel deep neural network archi- tecture known as Fast for the vertex coloring problem. Cummiin & Veeaas (0244 have highlighted the potential of RL to tackle the VCP, but also pointed out its limitations without inver-invariant representa- tions. They have emphasized the importance of integrating GNNs to enhance RL performance by providing essential structural insights. Gianinazzi et al. (2021) have proposed a greedy combined probabilistic heuristic for the VCP that. integrates reinforcement learning and an attention mechanism (Vaswani 2017) for vertex selection. In their framework, they only used a terminal step reward based solely on color count. Similarly, Watkins al. 2003) have introduced ReLCol, a method that combines Q-learning (Walkins & Dayan 1992)
with GNN for feature extraction.
The most closely related work is by Ahn et al. 2020) in which they propose the deferral action strategy for solving the Maxinum Independent. Set (MIS) problem. After evaluating various MDP configurations we find that the deferral action strategy is also effective for the bulk assignment (assigning colors to multiple vertices at a step) in VCP, providing fiexibility to defer decisions for certain vertices. Consequently, we incorporate this strategy into VCoIRL. The Vertex Coloring Problem (VCP) introduces several unique challenges for reinforcement learning compared to the Maximum Independent Set (MTS). These include increased model complexity due to larger action space, strong dependencies between partially colored and uncolored regions
3

---

Published in Transactions on Machine Learning Research (09/2025)
of the graph, and the presence of multiple equivalent solutions from graph-partitionnng perspective. A detailed discussion of these challenges and how we address them is provided in Appendix D.1
3 Framework for Vertex Coloring Problem
We now describe our framework for the VCP. Given a graph G = (V, W) with vertex set V and edge set W, the objective is to assign colors to the vertices of a graph with the minimum number of colors, such that no two adjacent vertices share the same color. In our approach using a multiclass vertex classification strategy with k color classes, selecting colors from the ordered set C (1,,,...... k) for each vertex. If some vertices remain uncolored due to timestep completion color insufficiency, they are addressed described in Section 3.3 The resulting solution can be represented as a vector x = [a, i e V] E ({*} U C)), where each element I; either indicates the color assigned to vertex from the set C, or that vertex has not been assigned any color (xq = +). For our experiments, k: = 15.
the VCP, we attempt to color the vertices
3.1 Markov Decision Process for the VCP
We model the VCP as a finite MDP, which terminates when either all vertices are colored or the time limit (episode length) reached. The key components of the MDP are:
State: In VColRT, the state the system at time is represented as vector [s, E VJ e ((+)U)) where 8; C denotes the color assigned vertex i, and s; indicates that the vertex undecided, meaning it is yet to be colored. At the beginning of episode (at t 0), all vertices of the graphs are undecided, i.e., = for all E V. An episode terminales when either all vertices are assigned a color the time limit is reached.
Action: Action represents the color assignment decisions that the agent takes for the undecided vertices by observing a state a of the system at any time I. Let V, denote the set of undecided vertices. We consider two models for action:
Model with deferral: In this model, the agent can either defer the decision for undecided vertex assign it color. This is represented by =a : E V.] E ((+JUC))V. The action allows the agent to postpone coloring certain vertices and instead focus on subgraph it considers important to solve first
Model without deferral: In this model, the agent must assign a color to each undecided vertex without the option to defer. This is represented by a. = (a i E V.] E CIKI.
The transition from state a to by taking action a, occurs two phases: the update phase and the clean-up phase.
Transition:
Update Phase: The action a, determined by the policy for the undocided vertices V. ; is applied to create intermediate state é. Specifically, == a, if E V., and s, s; otherwise.
Clean-up Phase: The clean-up phase ensures the resulting state s is conflict-free, For this, we consider two rollback strategies:
Hard rollback: Under this rollback strategy, all conflicting vertices are reset to the undecided state, irrespective of when they were colored, providing greater flexibility in terms of resolving conflicts.
Soft rollback: Under this strategy, only the vertices involved in conflict. during the latest. assignment reverted, leaving previous assignments unchanged.
The reward for transition s + s resulting from an action a. is a weighted sum of two types of reward, namely the verter satisfaction reward and the color usage penalty. Verter satisfaction reward accounts for the change in the number of assigned vertices between two states, i..., Sat(s') Sat(s), where Sat(s) Eiev I (s, E C), and I the indicator function. Color usage penalty encourages minimizing the number of colors, for which consider two strategies that impose the penalty in different. ways:
Rewards:
4

---

Published in Transactions on Machine Learning Research (09/202))
(8,-2)
Terminal
6
a=f1,2,1.22
2
(.4.2)
B
Hard Rollback
(4,-2)
a=(2,1]
a=11,2,11
2
Soft Rollback
(0.0)
2
(4,0)
Terminal
a=[1,2]
Figure Illustration of MDP for the VCP. The arrow represents the final transition, the black arrow represents the update phase, and the blue arrow represents the cleanup phose. The of elements in the tuples (verter satisfaction reward, color usage penalty) denotes the immediate reward for a transition, while the numerical value in the terminal state represents the total returns, calculated by summinn the rewards of all transitions in the episode. The vertices highlighted in yellow indicate the subgruphs where actions are taken, with the action vector a ordered by vertices A, B, C, and D, maintaining the same verter order within the subgraphs.
Mar-color strategy: Penalizes based on the highest-numbered color assigned from C, defined as:
UB(s) = 0. if =, V. max (8: 8: E VVViEV,, otherwise.
Color-cound stralegy: Penalizes based on the total number of distinet colors used, defined as
0, if = =, H& | otherwise.
Count(s)
For a transition s s', the immediate reward using the maz-color strategy is given by Ta-sa' wi-[Sat(s') Sat(s)] + W2 (() - (B(s)) )}, whereas for the color-count strategy, the reward calculated similarly, with the UB lunction replaced by the Cound function. Since our objective to ensure that all vertices are colored, we give more weight the vertex satisfaction reward, ie., w1 w2. For our experiments, we set W1 and Wy 1. During training, normalize the individual rewards before combining them by dividing the verter satisfaction reward by the number of vertices in the graph and the color usage penalty by the number of colors in set C.
Illustrative example of VCoIRL's underlying MDP: Figure illustrates the vertex coloring process for a graph with four vertices labeled A, B, C, and D. The initial state (,, indicating that no vertices have been assigned colors yet. In the initial state, an action [1, 2, *, is taken, assigning colors and 2 to vertices A and B, respectively, resulting in the next state [1, *, *]. This transition yields vertex satisfaction reward of w, 2 = 4, where wi 2 is the weight for vertex satisfaction, and 2 vertices are satisfied. Furthermore, the color usage penalty for this transition is w2 -2 = 22, where W2 1 is the weight for color usage, and 2 colors are used. The reward corresponding to vertex satisfaction and color usage for this transition is represented as a tuple (4)) and the lotal transition reward for this step is obtained by summing the values inside the tuple, yielding a reward of 2. Now, in the current state [1,2, *]. an action [2, 1] applied the undecided vertices C and D, resulting in an intermediate state of 21, which leads to conflict and depending on the rollback strategy used, the transition proceeds in different directions. In the hard rollback strategy, all vertices involved in the conflict reverted, resulting in
5

---

Published in Transactions on Machine Learning Research (09/2025)
[0,0,0,0]
[0,0,0,0]
[1,11,1,11
[0,0,0.0]
11,0,000]
[1,1,1,111]
[1,0,0,0]
[100000]
[0,0,0,0]
[0,0,0,0]
[1,1,1,1,1]
Feature update with OR oporation on neighbore' state vectors
Feature mapping and campling uncolored suD ud-ggrah
Figure Feature extraction and subgroph sampling.
state (+,,, &, *, and a final reward tuple (-4, 2). The action 2, is then applied in the current state [&, *, *, *I, leading to the final state 2, 1] with reward tuple (8, -2), giving total episode return (i.c., the cumulative reward in that episode) of 6. In contrast, the soft rollback strategy only reverts the newly assigned vertices and does not change the already colored part of the graph, resulting in the state 21, +, and a reward tuple (0, 0), indicating no additional reward or penalty from the reverted assignments. The action [1, is then applied in the current state [1,2, x, *], for vertices C and D, resulting in the final state 11,2,1, 1.2] with reward tuple (4, (), giving a total return of for the episode.
3.2 Model Training and Hyperparameters
Proximal Policy Optimization algorithm: We use PPO (Schulman et al. 2017 to train the agent solve the VCP problem. The objective for the actor is expressed as:
Lactor (0) = E. [min (r.(0)A,, clip (r,(0), 1,
Ko (a.le,) is the probability ratio of the new policy to the old policy at time 6, e a hyperparameter that defines the clipping range to prevent large policy updates, Åg is the advantage which represents the difference between the actual return and the estimated value of the state at time t. 0 represents the parameters of the policies. In addition, PPO also has critic loss, and entropy regularization term is added to the total objective to encourage exploration. For detailed explanation of PPO equipped with entropy regularization, refer to Appendix A.3.
where the subscript / denotes the time within the episode (Irajectory), r.( = Tonew (at[ss)
Model architecture and input feature: We use the GraphSAGE architecture (Hamilton et al. 2017) for the policy and value networks. This architecture assigns colors to the undocided vertices of the graph by considering their subgraph, and therefore, we design a verlex feature that effectively conveys all the necessary information the subgraph, enabling to identify which color assignments could lead conflicts. Consoquently, vertex feature vector is of the length of 1 [C], where |C| represents the number of colors in the set C. The first element of the feature vector is set to to reffect the vertex's degree after feature aggregation in the GNN (see Appendix A.2 The remaining |C|| elements represent the color usage status of the vertex's neighbors: a value of 1 indicates that none of the neighbors use the corresponding color, while -1 indicates that one or more neighbors use the color. To get the common vector representing colors used by the neighbors, we convert each of the neighbors' states into one-hot vector, with at the position corresponding the color it uscs, and then perform an OR operation on the neighbors' onc-hot state vectors. Figure 2 shows the feature extraction and subgraph sampling process of VColRL with |C| = 4. After sampling the subgraph, message passing, as it may lead to a loss of crucial neighborhood information. For details about the VCoIRI. architecture, GraphSAGE architecture, and PPO implementation, refer to Appendix A.I Appendix A.2 and Appendix A.3
do not perform random sampling within the new neighborhood during
Training dctails and hyperparamcters: We generate a graph dataset containing 15, 000 random Erdös-Réón (ER (Erdos al. 1960)) graphs with 50 100 vertices and an edge probability of 0.15. This data set. is divided into two parts: the first 14, 000 graphs are used for training, while the last 1000 graphs are

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Haperparameters used for training.
Hyperparameter
Value
Hyperparameter
Value
Episode length/ Time limit
32
Eutropy regularization coefficient
0.01
Replay buffer size
32
Learning rate of optimizer
0.0001
Batch size
32
Gradient norm
1
Batch size for gradient step
16
Discount factor for PPO
1
Number of gradient steps per update
Clip value for PPO
0.2
Critic loss coefficient
Message passing layers 0.25
4
used for validation. The dataset contains graphs with chromatic numbers ranging from 4 to 7. Both actor and critic nerkorks consist of hidden layers (number of messaging passing steps), each with a dimension of 128. ReLU is used the activation function. We train the model for 300 epochs using the ADAM optimizer (Kingma 2014p. The model configurations and hyperparameters are selected empirically with the help of the Optuna (Akiba et al.) 2019) framework, described in Appendix C.1 During the experiments, we observed that all MDP variants converged to the best objective for similar values of hyperparameters. Table ! presents the hyperparameters under which models are trained.
3.3 Addressing Incomplete Solutions
The model designed output color assignment for each vertex at each state, which requires learning probability distribution initial set of k colors. An incomplete solution may occur cither when the time limit is reached before all vertices are colored, resulting in fewer than l colors being used, or when all k colors are utilized but some vertices remain uncolored. In either case, the subgraph of uncolored vertices is extracted, and the multiclass vertex classification restarted fresh on the subgraph. If, in the previous iteration, & colors were used, the current iteration starts with color l + 1. This iterative process continues until all vertices are successfully colored. The final color count is obtained by summing the total number of colors used across all iterations. For our experiments, k 15.
fixed set of colors. VColRT begins multiclass vertex classification with an
4 Performance Evaluation of VCoIRL
In this section, we first compare different MDP configuratioss to identify the best-performing one for training. We then evaluate VColRT, trained with this configuration, against several baselines across diverse families of benchmark and synthetic graphs to assess its effectiveness in solving the VCP, as well as its generalizability and scalability. We use an NVIDIA GeForee RTX 4090 GPU for training models, and a 12th Gen Intel0 CM 17-12700 processor featuring 20 logical cores with 32 GB RAM for testing models and baselines.
4.1 Comparison of MDP Configurations
We analyze the validation results of models based on all the combinations of rollback strategies (soft rollback and hard rollback), action types (with deferral and without deferrul), and reward strategies (mar-color and color-count), resulting in a total of 8 configurations. The evaluation is conducted on 1000 validation graphs, enabling thorough assessment to identify the optimal MDP configuration.
Figure 3a shows the return (sum of rewards) over epochs for validation graphs. Figure 3b illustrates the average number of colors used, reflecting how color usage evolves during training. Figure 3c presents the graph satisfaction percentage, indicating the proportion of validation graphs where all vertices are satisfied in one MDP trajectory. Together, these figures offer a comprehensive view of the agent's decision-making process, highlighting its impact on color efficiency and graph satisfaction. In addition, Table summarizes the results of these figures.
Figure 3a demonstrates that returns increase for all MDP configurations, indicating that the agent effectively learns to fulfill its objective. Similarly, Figure 3b shows consistent decrease in the number of colors used
7

---

Published in Transactions on Machine Learning Research (09/202))
Average colors used
14
HNNM
HDM
HDC
Graph satisfaction (%)
100
Average retum
1.4
NAREY
SWI
SWC
1.2
10-
60
1.0
HDC
HWM
HWC
HWM
HWC
0.8
SIM
S0C
50C
SWC
SWC
100
200
100
200
300
300
Epochs
Epochs
Epochs
(a) Returns
(b) Colors used
(c) Graph salisfaction
Figure 3: Performance of VColRL with different MDP configurations. The strategies used in our framework are represented by the followiing abbreviations: H denotes hard rollback, S denotes soft rollback, D denotes with deferral action, W denotes without deferral action, M denotes maz-color reward strategy, and C denotes color-count reward strategy, e.g., HDM denotes hard rollback with deferral action and maa-color reward strategy.
Table 2: Summurry of Figure Maæ Return indicales the highest average relurn achieved across validation evalua- tions, Min Color Used denotes the minimam, of the average colors used across validation evaluations, and >> 95% Graph Satisfaction refers to the number of validation evaluations which the model completely colors more thon 95% of the graphs.
Configuration
Max Return
Mín Color Used
95% Graph Satisfaction
HDM
1.59
6.155
282
HWM
1.57
6.347
SDM
1.56
6.445
SWM
1.57
6.379
HDC
1.58
6.218
195
HWC
1,58
6.247
80
SDC
1.58
6.203
SWC
1.57
6.337
across epochs for all configurations, reflecting the agent's ability minimize color usage, Among these, the HDM (hard rollback with deferral action and maz-color reward strategy) emerges the best-performing configuration, achieving higher returns and using fewer colors average. This is particularly evident between epochs 150-300, where the blue line representing HDM stands out by being above others in returns and below others in average colors used. In Figure 3c. all hard rollback configurations initially start with lower graph satisfaction values, gradually increase to 100%, but differ in their post-convergence behavior. While HDM maintains full satisfaction, demonsttatingg stability, other configurations begin to deteriorale, resulting instability. In contrast, the soft rollback configurations start with high graph satisfaction (near 100%), experience significant drops, and then oscillate before recovering to approximately 90% in 300 epochs, implying slower convergence and longer training times. Not maintaining graph satisfaction indicates that the agent fails to color all the vertices within the allotted time limit. Despite this, Figure 3a shows that returns remain relatively stable for such configurations, suggesting that only few vertices per graph are left uncolored; however, this undesirable since we give more weight to the verter satisfaction reward as described Section 3.1 Overall, HDM consistently achieves full graph satisfaction along with minimized color usage, demonstrating its ability to color graphs more efficiently within less time. This is also evident from Table where the HDM configuration achieves higher average returns, uses fewer colors, and maintains over 95% graph satisfaction in 282 out of 300 validation runs, which is better than all other configurations, To further support our claim, we run each variant five times and present the results in Appendix C.4 As shown in Figure 7 and Figure g the HDM configuration consistently yields more stable policies in terms of both graph satisfaction and color usage compared to the other variants.
HDM emerges as the best configuration due several key factors. First, the hard rollback mechanism in HDM reverts all conflicting vertices the undecided state, unlike soft rollback, which only reverts the newly assigned vertices while leaving the previous assignments unchanged. Since reinforcement learning
8

---

Published in Transactions on Machine Learning Research (09/2025)
relics trial and error, hard rollback offers greater flexibility, enabling the model to explore better solutions more effectively. Additionally, the with defernal action strategy allows the agent to break the problem into simpler subproblems by postponing some decisions to later timesteps. This mirrors human problem-solving, where certain vertices are addressed first to simplify the decision process in subsoquent stages. Conversely, in the without deferval action stralegy, the agent is forced to make decisions immediately, regardless of its confidence, which hinders its performance. Lastly, the maz-color reward strategy treats the color set as ordered, encouraging the agent to assign colors sequentially from the first color up k, ensuring the solution is unique in terms of the colors used. This contrasts with the color-count strategy, where multiple optimal solutions can exist. For instance, if three colors are needed to color a graph, under the color count strategy, both color assignments '1, 2, 3" and '1, 2, 4' will yield a color usage penalty of 3, whereas, under the mar-color reward st stralegy, the former will yield a color usage penalty of 3, while the latter will incur a penalty of 4, ensuring a unique and contiguous solution in terms of used colors in the trained model thereby roducing randomness in the learning process by providing an organized approach for the VCP.
In summary, HDM is an MDP configuration developed by combining our novel hard rollback mechanism and maz-color reward strategy with the defervol action strategy proposed by|Ahn et al (2020). This configuration outperforms others in terms of training time and graph coloring efficiency, making it a suitable choice for VCoIRL.
4.2 Comparing VCoIRL against Baselines on Diverse Graph Families
We now compare the VCoIRL model trained on the HDM configuration against several basclines across diverse families of benchmark and synthetic graphs. Performance is evaluated based on the number of colors used and execution time, i.e., the total time taken by the algorithms until termination. Due to the stochastic nature of the model, for the graphs with more than 10, 000 vertices, we integrate diversification search engine into the model. To do this, we batch 100 disconnected instances of the graph to form a single graph, which is then fed into the model. After obtaining the output, we search for the best solution. In such cases, the reported execution time is inclusive of all these processes.
Baselines: For our experiments, we compare the performance of VCoIRL against six baselines. The first baseline, First Fit (FF) Gyárfás & Lehel 1988), is a simple greedy approach that processes vertices soquentially and assigns each the smallest feasible color that does not cause a conflict. The second baseline is TabucolMin, enhanced version of the TabuCol (Hertz & Werra 1987) (a metaheuristic search algorithm), incorporating an additional minimization step to refine the coloring. The third baseline is VCoIMIS, which is based on the Maximum Independent Set (MIS) reduction approach. Starting with the input graph, we iteratively compute an MIS using the model proposed by (Ahn et al. (2020). assign a new color to all vertices in the set, and then repeat the procedure on the subgraph formed by the remaining uncolored vertices until all vertices are colored. The fourth baseline is the Gurobi 12 solver (Gurobi Optimization, 2024). a state- of of-thc-art optimization solver. For this bascline, we formulate the VCP as integer lincar programming problem. The fifth baseline is FastColor (Lin et al. 2017). ; baase heuristic search method based on Bounded Independent Set (BIS). Lastly, we include ReLCol (Warkins et al.] 2023). a heuristic for the VCP based on Deep Q-Networks (DQN) and GNN. For this, we directly use the trained model provided by the authors. Further details about these baselines are provided in Appendix B Gurobi and FastColor are search-based baselines that run until either optimality proven or a user-specified cutoff time is reached. If optimality is not established within the cutoff, the reported exccution time is set to the cutoff, since the baselines continue to consume resources until termination. The cutoff time for these baselines is tuned with respect to the benchmark graphs used for evaluation, as detailed in Appendix C.2 and the execution time reported for these baselines is based on these cutoffs.
Performance on COLORO2, DIMACS, NR and SNAP benchmarks: We now compare VColRL against baselines on a subset of the COLORO2 and DIMACS (Trckk 2002-2004), NR (Rossi & Ahmed 2015) and SNAP (Leskovec the range of 4 8, aligning with the 4 7 range of the training dataset. For Gurobi, the cutoff time is set to 900 seconds, while for Fast Color, the cutoff time is set to 60 seconds, used in the original paper.
Krevl, 2014) benchmark datascts, focusing on graphs with chromatic numbers
9

---

Published in Transactions on Machine Learning Research (09/2025)
Table 3: Performance comparison across the COLORO2 and DIMACS benchmarks [1/2]. Each entry in the table contains either siæ two values. Entries with sie values report the rummber of colors used (best solution) and execulion time (in seconds), followed by the mean and standard deviation of these Iwo quantilies across a hundred runs. Entries with only two values omiil the mean and slandard devialion due to the algovitlan being either determoinistic or the standard deviations being zero The best- pert-perfornnn algorithms, in terms of minimizing the number of colors, are boldfaced for each graph.
Graph Type Varcion Edggss
FF
Mib
VCaIMIS 6. 0.61 6.06, 0.31, D.88, 0.12
Curoli
ReTCol
VOGIR 0.29 5.36, 0.94, 0.09, 0.09 0.05 6.70,
wIdIIGPIA
662, 4181
10, Te
4. 1111.02
$13.35
4. 61.000
T, 252519906
mARnSGPLA
1206, 7844
9, 1
254.72
0.77 6.08,
0.76, 0.07
900000
4. 60,,00
T. 10000.91
0.11, 0.11
1916, 12506
10,
500.98
6,12, 0.32, 0.57, 0.13
900.00
4.52, 0.50, 00.00, 0.04
T, 0.60771.44
0.50, 1.3i, 0.15, 0.12
1. Palllns 3
50, 100
8, 3e-5
4, 1.00
4.112 0.20, 0.26, 0.07
4. 0.35
4. 0.51
4. 00,000
4.0.26
4.73, 0.44 0.01, 0.005
4. 0.01
1-Piillus__ 4
90. 203
11. 16
5.05, 0.26, 0.51, 0.06
0.19
G. 20.78
5. 601.00
5. 5.00
5.00,
0.03
0.03, 0.01
1-Fillho 5
252 32 07
14,
8.81
6.01, (1225, 0.41, 0.11
0.22
900.00
6. 60.00
8, 150.15
0.21, 0.00 OL1T, 0.07
2-Pullinx 3
52 201
10,
5, 0.91
3.01, OX (.ED, 0.51, 0.07
0.42
5, 0.72
5. 60.00
5. 1.100
5, 0.01 5,00, 0.25, 0.03, 0.03
2 2-Fulllne_ J
212 1621
14,
2.5
6.05, 0-40, 0.30
0.58
6, 82:75
00.00
59.80
0.12, 0.21, 0.10
0.19
2-Fellho 5
852 12301
18,
7. $4.10
7.05, 0-47, 0.50, 0.13
0.54
900.00
7. 60.00
5181633
7:000 1.56, 0.24, 0.10
3-Felline_ 3
60 316
12, 76
6, 1.04
6.01, 0.10, 0.51, 0.10
0.52
6, 3.90
6, 00.00
6, 6.777
6. 0.30 6,57, 0.59, OLIS, 0.08
3 Fullins J
405 3524
17. 64
4.45
7.05, 0.21, 0.82, 0.11
0.72
5.33322
7. 60,00
8. 524.2777
7.53, 0.18. 0.08
4 a
114.
14,
7.07, 0.25, 0,80, 0.11
0.58
5.50
T. 60,00
7. 7:32
8.20, 1.7A, OLGO, 0.18
4-Fulliox_4 4
600, 6630
20, Ge-4
8, 16.01
8. 0.58
8. 900.00
8. 611.00
9, 2519999
8, 0.65
8.11 0.34, 0.75, 1.10
10.36, 1,8, 0.74, 0.19
5 Pulllas 3
154, 792
18, 10
1.14
8.02. 0.14, 0.75, 0.30
1.01
8. 5.96
8. 60,000
20.51
11:21, 1.71, 0.37. 0.13
0.29
L-Imertions__
67. 212
5, 5
5.01, 0.10, 0.19, 0.05
0.29
5, 31.07
5. 60,00
5. 1.66
5.01, 0.10, 0.02, 0.02
1 Inneertoons_ 5
202 1227
6, 24
6, 1.24
6,01, 0.10, 0.20, 0.05
0.31
6, 900.00
6, 600,00
46.70
6, 0.36 6.30, 0.44, 0.10, 0.08
IInsettics__ 6
607 6037
7. 7c
7. 32.76
7,07, 0.50, 0.11
7. 0.18
900.000
T. 600000
1888.44
7.54, 0.19, 0.11
7. 0.16
2-1mertion 3
87, 72
4,
0.47
0.12
0.44
4. 60,00
4. 0.030
4.58, 0.95, 0.01, 0.005
2-neertionn_
149 541
5, 14
5, 0.73
5.0S 0.27, 0.22, 0.08
0.19
5, 900.00
5, 60.00
5. 16-71
5.01, 0.1. 0.01, 0.000
5, 0.02
2 Inneertions_
597 3035
6,
6.5L
6,500 0.54, 0.11
0:30
900.00
60.00
1636.20
6.35, 0.10, 0.09
0.12
3-hmertion 3
56
4,
0.50
0.05
4. 5.06
4. 60.00
4. 4.0.7
4.50) 0.50, 0.01, 0.00
3- I-neertionn_ 4
251 10:00
5, 2-
5, 1.25
5. 0.22 5.09, 0.12, 0.20, 0,10
5, 900.00
5, 60.00
5, 141.07
5, 0.00
3 Insertions__
1406, 9895
6, 1e-3
6, 35,57
6,90, 0.42, 0.30, 0.15
0:31
900,00
60,00
6, 19407.82
6,30, 0,20, 0.17
0.10
Insertions_
79.
4.
4.01, 0.10, 0.20, 0.07
4. 14.71
4. 60,00
4. 2.00
4.76, 0,42, 0.01, 0.005
4-1mertions. .4
476. 1795
5, 3c-4
2.23
5.10, 0.30, 0.29, 0.12
0.25
900.00
6. 61.00
5. 791.63
5.00, 0.20, 0.02, 0.03
5, 0.02
For COLORO2 and DIMACS benchmarks, the results are reported Table 3 and Table 4 We observe that VCoIRL performs equivalently or better than FF, VCoLMIS, and ReLCol in terms of using fewer colors across all graphs. It outperforms these basclines on ~ 53%, 23%, and ~ 37% of the graphs, using an average of 6.73, 1.70, and 3.43 fewer colors, respectively. Compared to Gurobi, TabucolMin, and FastColor, it performs equivalently or better on ~ 95%, ~ 86%, and ~ 86% of the graphs, respectively. It outperforms Gurobi on ~ 5% graphs and TabucolMin on ~ 2% graphs, using, on average, 3.28 fewer colors than Gurobi and 2 fewer colors than TabucolMin. In terms of exccution time, it is up to 10 faster than its RL competitor ReL,ool, ~ 40x faster than VColMTS, and ~ 20000x faster than TabucolMin on some graphs. For baselines with a cutoff timer, it achieves specdups of up to ~ 45000x over Gurobi and ~ 6000x over FastColor on certain graphs under the chosen cutoff.
Similarly, for SNAP and NR benchmarks, the results are reported in Table 5 We observe that VCoIRL performs cquivalently or better than FF, VCoIMIS, and ReLCol in terms of using fewer colors across all graphs. It perporrsss these baselines on ~ 80%, ~ 60,, and ~ 42% of the graphs, using an average of 2.66, 2.88, and 3.33 fewer colors, respectively. Compared to Gurobi, TabucolMin, and FastColor, it performs oquivalently or better on ~ 93%, ~ 93%, and 87% of the graphs, respectively. It outperforms Gurobi and TabucolMin on- 47% graphs, using, on average, fewer colors than Gurobi and 2. fewer colors than TabucolMin. In terms of execution time, it is up to 50000 faster than its RL competitor ReL,Cll, ~ 36x
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 4: Performance comparison across the COLORO2 and DIMACS benchmarks [2/2]. Each entry in the table contains either siæ two values. Entries with sie values report the rommber of colors used (best solution) and execulion time (in seconds), followed by the mean and standard deviation of these Iwo quantilies across a hundred runs. Entries with only two values omiil the mean and slandard devialion due to the algovitlan being either defermeinistic or the standard deviations being zero. The best- pest-perfoniin algorithms, in terms of minimizing the number of colors, are boldfaced for each graph. Empty entries indicate the algorithm did not run due RAM limitations.
Graph Typu
Varliew, Edgas
FF
TabunalMiia
VCGIMIS 9, 1.088
Gurohi
FuutColue
Rabbol
VCaIRL
In100 5t
450, 071d
14, Te
6, 510.47
10.82, 0:50, 1,31, 0.35
7. 900000
7.14, D.SI, 18.98, 5.30
5, 55.05
15, 7:1.41
7.48, L.7, 0.33, 0:20 G. 0.08
6. 0:29
1e4b0__5b
450 5734
10, 7o
5, 645.1.7
9. 0.88 11.06,
0.20
9. 903000
5. 50.08 G0L60, 0.58
13. 729,42
7.55,
le
450, 9808
17,
663,22
9:21, 0.95, 0.02, 0.19
7,
7. 900000
53, 0.96, 53.00, 22.60
9.20
16, 725.36
0.00, 1.26, 0.15, 0.10
Iw100 5d
150 9757
18, 1x
7. 59.22
9.10, 0.01; 0.63, 0.90
7, 0.188
13, 900.00
8, 12.51
17, 736.01
0.47, 1:26, 0.27, 0.20
5, 0.06
146
4.13, 0.33, 0.40, 0.04
4, 0..0
4. 3.00
3.56
4.96,
0.10
mmn088 25
88, 146
4.31, 0.SI. 0.89, 0.04
4. 11.20
4, 3.05
00.00
3,5.5
4,90, 0.14, (103, 0.01
0.00
mug/00011 1
100, 100
4. B
4, 0.53
4.04, 0.19, 0.49, 0.03
4, 0.49
4, 0.69
4. 00.00
4. 5.541
4.50, 0.1. 0L06, 0.02
4. 0.54
mmg100022 25
100, 110
0.50, 0.49, 0.04
4, 0.47
0.47
4. 00,00
4. 4.4.9
5.00, 4. OUUCC 4.233, 0.82, (102,, 0.01
0.02
mpceit3
11,20
4,
4, 0.19
4, 0.01
00.00
4,003
mycield
25,71
5, 2e
5, 0.63
5, 0.15
5, 0.52
5, 00.00
5,0.4
5,02, 0.2, 0.04, 0.06
5, 0.02
mnycielG
235
6, 0.34
6, 6.74
00.00
6,0.74
0.27, 0.70,
0.07
mygist0
95, 755
7.
1:20
7:24, 0.42, 0.29, 0.08
7, 0.11
7. 900.00
7. 00.00
5.333
7:15, 0.500 0.26 0.18
0.06
myciel?
191, 2000
8, de
4,877
8.20, 0.42, 0,32. 0.10
8, 0.14
8. 900.00
8, 90.00
8, 1.42225
9.51, 0.8L, 0.12, 0.02
W. 0.57
quaee6_B
100
8, 4e
5.91, 1.01, 0.27,
5, 0.11
5, 0.11
0.000
5,0.1
5,29, 0.50,
5, 0.10
queent. 6
36, 200
11, de
7. 4.91
9.20, 0,85, 0.69, 0.15
8, 0.39
7, 0.39
7. 00.00
8. 8.0.4
10U1,, 1,16, 0.53, 0.06
0.21
queen 7
49, 476
10, 8e
7. 14.85
10.30, 0.70, 0,77, 0.15
8. 0.53
0,0.3
7. 1.20
9,0885
12.97, 1.56, 0,55, 0.18
7,0.2 0.27
DRJCI
5, 12.58
1:50, 0:62, 0:72,
23.36
00.00
7, 11.10
0.30 0.62,
willieGPPA
201, 6772
11, 96
7. 19.06
10.05, 0.84, 1.50, 0,58
8, 2.55
9, 5.00000
7. 00.00
9, 32.8.72
10.20, 1.50, 0.32, 0.05
0.23
faster than VCoIMIS, and ~ 600x faster than TabucolMin on some graphs. For baselines with a cutoff timer, it achieves speedups of up to ~ 1000x over Gurobi and ~ 80x over FastColor on certain graphs under the chosen cutoff.
Performance on synthetic graphs: We now compare VCoIRL against baselines on different types of synthetic graphs comprising ER (Erdos et al. 1960), BA (Albert & Barabási, (002). and WS (Watts & Strogatz 1998) graphs, and the results are reported in Table 6 We evaluate 500 random graphs for cach row in the lable. We set the cutoff timer to 10 seconds for Gurobi, as would otherwise continue searching for optimal solutions for an extended period, making it impractical to evaluate a plethora of graphs within a reasonable timeframe. The cutoff timer for FastColor is set to 60 seconds.
We observe that for synthetic graphs, VCoIRL outperforms FF, VCoIMIS, and RelCol in terms of color usage across all graph types and ranges, while using on average 1.51, 2.23, and 1.95 fewer colors, respectively. It achieves competitive performance compared to Gurobi, TabuColMin, and FastColor in terms of color usage, using less than one additional color on average. However, it offers a significant advantage in execution time, being on average ~ 23x faster than its RT. competitor ReLCol, ~ 1.5x faster than VColMTS, ~ 13x faster than TabucolMin. For basclines with a cutoff timer, is on average ~ 14x faster than Gurobi and 60x faster than FastColor under the chosen cutoff, thus demonstrating good balance of efficiency and performance.
Generalizability: From Table 3 Table 4 and Table 5 we observe that VCoIRL outperforms or matches the best-performing baseline on 50 out of 58 graph instances (86%) in terms of color usage. On the graphs where VCoIRL slightly underperforms, it requires only 1.125 additional colors on average compared to the best bascline. This minor gap demonstrates the strong generalization ability of VColRL across diverse graph types and sizes, as the model trained exclusively on ER graphs performs well on test graphs from variety of unseen distributions. To further validate this generalization, we compare VCoIRL with model trained
11

---

Published in Transactions on Machine Learning Research (09/2025)
Table 5: Performance comparison across the NR and SNAP benchmarks. Each entry in the table contains eilher six or two volues. Entries with six values report the number of colors used (best solution) and execution time ('n seconds), followed by the mean and slandard deviation of these two quantilies across a hundred runs. Entries with only two values umit the meun and slandard devialion due to the algorithm being either deterministic or the slandard deviations being zero. The best- perfoorrfonnnn algorithms, in terms of minimizing the number of colors, are boldfaced for each graph.
Gruph Type
Varticus, Edggon
FF
MabunalMi
VOGIMIS 5. 0.53
Guroli
Calo
ReLOOl
VCaIRL 6. 0.09
inceuaily
6800, 7680
5, 2m-4
14, 30.08
6, 23.33
5, 0:21
5.74, 0.64, 0.50, 0.08
5,30, O.7. 0.33, 0.11
lo Eb messages
1206, 6341
15, 10
8, 35.02
11.77 1.31, 0.14
10, 1.21
11, 902.00
6, 120000
13648.69
9. 0.72 31.00, 0.76, 0.00
p4p-Chutella4
10579, 83094
8,
1158.90
8.97, 0:41, 25, 82, 0.36
25,57
14, 902.00
5.3, 0,45, 50,000 0.21
5. 80.00
6.95, 0.43), 18.90, 0.21 6. 25. 5.06, 0.201, 25.27, 0.21
S. 16,05
pdp Gautella24
21318, 63300
9, 20-
6, 3640.00
6,50 0.12, 43.66 0:42
8, 43:42
15, 902000
5. R0.00
Gantella25
22087, 54736
8, 1e-1
6. 2220.00
8.17, 35. 0.30
55.30
15, 902.00
62.00
22.12
[ap-Gmitolle30
56052, 85328
8,
7319.00
8.03, (-10, 53.95, 0.58
5L.88
10, 903.00
5, 00.00
5.20, 0-49, 30,97, 0:35
$1.49
pop-Gootella3
67500, 147882
8, 4e-
6. 29700.00
9.08, 0.27, 90 2.51
9, 95.18
15, 903.00
5, 00.00
5.02, 0.14, J9 LS, 0.47
I. 49.05
117
5, Se
0.82
0.35,
4. 0.19
0.004
4. 3.81
0.36, 0.00,
0.01
761, 1029
7, 75
5, 5.1.1
7:63, 0.56, 0.80, 0.12
0.70
4, 27.12
4. 0.00
7, 7,900580
5.60, 6.65, 0.0,, 0.06
0.05
sot-Colphins
62, 150
7, 76
5, 1.57
6.06, 0.29, 0.00, 0:21
5, 0.44
5. 0.01
5, 0.00
6. 1.30
6, G.0L 5.11, 0.34, D.06,, 0.02
suc-karate
34, 78
6, 4m
5.00, 0.00, 0.10
5, 0.35
0.008
0.003
5, 0.001
5.03, 0.00,
6. 0.01
son-eriki-ootee
850, 2914
10,
7, 23.58
12:27, 0.72, 1.87, 0.17
11, 1.25
96.08
7, 0.42
12, 45
11,46, 1.25, 0.74, B.28
0.40
Eio-yauo
1458, 1048
6, 7c
6, 2,50
6,13, 0.33, 0.85, 0.06
6, 0.85
6, 106.32
6, 1.07
6. 12778.50
7.51, 1.78, 0:25, 0.12
6. 0.24
inf-punor
1941, 6594
2m-3
0, 28.12
6,82, 0,51, 1.00,
6. 1,06
171.27
0.22
6. 0.81 6.35. 0.89,
todi pedd-pp--guutelll clla
02661, 147H78
8, 4e
8. 20197.00
9.02, 0.14, 08.27, 1.11
101:25
15, 902000
5,02, 0.34, 60.00, 2.25
5, 00.00
5.02, 0.34, 40.28, 0.19
19.00
Table 6: Performance comparison across different synthetic graph types and verter ranges. Each entry in the table has two values: the average number of colors used, and the average execution time in seconds. The objectives of the best-pet rest-performing methods in terms of minimizing color usage are boldfaced. In ER graphs, p is the probubility of edge creation; in BA, n is the number of edges to attach from a new verter to eristing vertices; whereas in WS, n is the rumber of nearest neighbors to which each vertez is joined within a ring topology, and p is the probability of rewiring each edge.
Graph Type
Vertex Range
TabueolMin
VColMIS
Gurobi
Cast.Color
ReLCol
VCoiRI.
ER (p=0.16)
50-100
7.648, 86-3
5.252, 5.55
7.216, 0.57
5.322, 5.02
5:264, 53.05
6.294, 2.83
5.502, 0.36
ER (p-0.125)
100-150
9,330, le
6.240, 10.87
8.688, 0.74
6.674, 10.00
6.492, 60.00
9.056, 11.90
6.760) 0.65
ER (p-0.10)
150-200
10.054, 2n" 4
6,554, 15.53
9,274, 0.86
7.340,
10.00
6.958, 60.00
10.140, 33.67
7.528, 0.80
BA (n=8]
50-100
6.682, 8e
6.110, 3.39
9.054, 0.48
6.008,
4.06
6.008,
21.53
7.100, 2.88
6.108, 0.46
BA (n=8)
100-150
6.866, 1e-4
6.238, 4.55
9.960, 0.57
6.028,
8.41
24.65 6.028,
9.054, 11.55
6.686, 0.51
BA (x=6)
1501-2000
6, 6,920, 1e
6.412, 6.38
10.412, 0.62
6.030,
9.94
6.030,
26.55
9.858, 32.99
6,898, 0177
WS (m=15, p-330)
50-100
8.778, le
6.574, 7.63
5 8.604, 0.67
6.838,
7.52
31.78 6.678,
7 7,986, 2.82
6.996, 0.54
WS (x=15, p=030))
100-150
8,968, 2e -
10.11 6.126.
8.716, 0.72
7,001,
10.00
37.17 6.964,
206., 12.06
7,008, 0.70
WS (m=15, p=0.30)
150--200
9,038, 2e
6.490, 11.74
8,801, 0.90
7,022,
10.00
7,006, 35.64
9.512, 33.69
7,014, 0.82
on mixed dataset comprising ER, BA, and WS graphs. Both models achieve comparable performance, showing that VCoIRL generalizes effectively, even when trained on a single distribution. For detailed results, see Appendix C.3
Scalability: From Table 3 Table 4 and Table 5 we observe that VCoIRL exhibits good speed advantage over all baselines, except FF, which is the most basic heuristic with poor performance in terms of color usage. For some graph instances, it is up to 45000x, ~ 20000x, ~ 6000x, and ~ 40x faster than Gurobi, TabucolMin, FastColor, and VCoIMIS, respectively. Compared to ReLCol, which is also an RL-based bascline that uses GNN, VCoIRL is ~ 120000 - 1000000x faster on certain graphs such as 3- s-Insertions_ 5 and ash958GPIA. Despite both methods being RL-based and GNN-driven, the performance gap arises from architectural differences. ReLCol employs a complex model that processes the entire graph at each step and colors vertices soquentially, whereas VCoIRL, adopts a reduction-basor neural model that focuses on uncolored subgraphs and colors multiple vertices in bulk at each step. The reduction in VCoIRL is enabled by carefully designed feature vector outlined in Section 3.2 that effectively conveys information about
12

---

Published in Transactions on Machine Learning Research (09/2025)
seegeeee Table 7: Comparison between VColRL and sequential models with defer (SD) and without-defer actions (SW) in terms of best solutions on COLORO2 and DIMACS benchmarks. Each entry in the table has two volues: the number of colors used (best solution), and the erecution time (in seconds), The best performing models are boldfaced.
Graph Type
SD
SW
VCoLRL
Graph Type
SD
SW
VCoLRL
4Ph33IGPIA
6, 4.08
7, 7.74
5, 0.29
3- Inseertons__
6, 9.69
22.22
0.16
ash60sGPIA
6, 13.69
31.72
0.04
4-Insertions
0.38
1.74
0.01
as1958GPIA
29.24
27.30
4-Insertions
3.66
3.67
0.02
1-Fallhns__
4, 0.10
4,
le450 5a
14.
13, 7.14
0.23
1-FullIns
5, 0.36
5, 0.71
5, 0.03
le450 5b
13, 10.68
12, 8.79
0.08
I-FullIns
6, 1.67
6, 4.03
6, 0.10
le450 5c
9, 6.24
12, 8.43
5, 0.04
2-FullIms
5, 0.23
5, 0.15
5, 0.04
le450 5d
10, 8.15
10, 5.37
5, 0.08
2-Fulllms
6, 0.94
6, 1.69
6, 0.49
mug88 1
4, 0.27
6. 0.89
4, 0.03
2.FullTns__
7, 3.93
9. 14.87
7, 0.06
mug88 25
4, 0.59
5. 0.93
4, 0.03
3-FullIns_
6, 0.35
6, 0.77
6, 0.10
mug100 (00
4, 0.72
6. 1.04
4, 0.04
3-FullIns
7, 4.58
4.50
7, 0.23
mug100 25
4, 0.69
6, 1.04
4, 0.02
4-Fulllus__
7, 2.44
1.09
0.17
mycie13
4, 0.13
0.10
4, 0.009
4-Fulllms 4
8, 8.34
10, 22.25
8, 0.65
mycic14
5, 0.16
0.22
0.02
Fulllns
9, 1.66
myciel5
6, 0.39
0.07
Insertions
5, 0.200
myclelf
2.17
0.06
Insertions
G, 0.89
7, 2.17
6, 0.10
myciel7
8, 4.66
11, 3.67
0.32
Insertions
8, 7.14
9, 8.93
7, 0.16
queen5
5, 0.29
5, 0.07
0.03
2- Iuscrtions
4, 0.10
4, 0.34
4, 0.01
queen6
8, 0.48
9, 0.42
7, 0.21
2- Insertions
5, 1.01
6, 1.50
5, 0.02
queen? 7
10, 0.92
9. 0.60
7, 0.27
Insertions
6, 3.84
7, 7.89
6, 0.12
DSJC125.1
8, 1.35
7, 1.26
6, 0.10
3- Iusertions
4, 0.33
4. 0.54
4, 0.01
will199GPIA
11, 10.56
11, 9.02
0.23
3- Insertions
5, 1.99
5, 2.54
5, 0.03
the colored portion of the graph to the reduced subgraph. To highlight the impact of the bulk assignment strategy in VCoIRL scalability, we now compare it with two sequential variants: Sequential-Defer (SD), (SW), which does not. In these variants, the model randomiy selects vertex to update at each step. The results in Table 7 show that VColRL requires an average of 5.60 colors, compared to 6.55 and 7.06 for SD and SW, respectively, demonstrating good generalizability terms scalability, VCoIRL is 2 500x faster than both sequential variants due to its bulk assignment strategy. Between the two sequential approaches, SD is typically 1-3x faster than SW.
which incorporates a deferral action, and Scquential-Withouu-
5 Conclusion and Future Directions
VCoIRL is a reduction based noural model built on a novel MDP formulation, termed HDM, which col- ors multiple vertices at each step, employs a hard rollback mechanism to resolve conflicts by reverting all conflicting vertices to the undecided state, and introduces a max-color reward that penalizes assignments based the highest indexed color used, thereby reducing label ambiguity and encouraging compact color assignments. Empirical results proves color usage over several baselines, including state-of-the-art solvers such as Gurobi, learning based approaches like ReLCol, and classical heuristics such as First Fit (FF) and VCoIMIS, in terms of color usage, while remaining competitive with metaheuristic and heuristic searches such as TabuColMin and FastColor, requiring, on average, only one additional color cases where underperforms. In terms of runtime, it is faster than most basclines, being up to 10 faster than its RI competitor ReLCol on certain graphs. Overall, the experimental results demonstrate that VColRL achieves strong scalability through its bulk as- signment strategy and exhibits robust generalization across graphs of different sizes and families, despite being trained only on small Rénni graphs with 50 to 100 vertices.
wide range of benchmark and synthetic graphs show that VCoIRL im-
Future research may further enhance the VCoIRL framework in both training and inference. One promising direction is the integration imitation learning, where high-quality heuristic solutions (e.g., FastColor (Lin) ct al. 2017)) can be exploited guide and accelerate training. Another the incorporation of problem- specific local search methods at inference time, using the learned policy as a prior refine solutions,
demonstrraed successfully in related combinatorial problems such MIS (Ahn et al. 2020 Feo et al. 1994]
13

---

Published in Transactions on Machine Learning Research (09/2025)
References
Shamim Ahmed. Applications of graph coloring in modern computer science. International Journal of Computer and Information Technology, 3(2):1 7, 2012.
Sungsoo Ahn, Younggyo Seo, and Jinwoo Shin. Learning what to defer for maximum independent sets, In International conference on machine learning, pp. 134-144. PMLR, 2020.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD Interna- tional Conference on Knowledge Discovery and Data Mining, 2019.
Saif Al-Sultan, Moath M Al-Doori, Ali H Al-Bayatti, and Hussien Zedan. A comprehensive survey on vehicular ad hoc network. Journal of network and computer applications, 37:380-392 2014.
Réka. Albert and Alszrt-László Barabási. Statistical mechanics of complex networks. Reviews of modern physics, 74(1):47, 2002.
Kai Arulkumaran, Mare Peler Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE signal processing magazine, 34(6):26 38, 2017.
Dimitris Bertsimas and John Tsitsiklis. Simulated annealing. Statistical science, 8(1):10-15, 1993.
Chris M Bishop. Neural networks and their applications. Review of scientific instruments, 65(6):1803-183) 1994.
Immanuel M Bonze, Marco Budinich, Panos M Pardalos, and Marcello Pelillo. The maximum clique problem. In Handbook of Combinatorial Optimization: Supplement Volume A, pp. 1-74. Springer, 1999.
Gregory J Chaitin. Register allocation & spilling via graph coloring. ACM Sigplan Notices, 17(6):98-101. 1982.
Gregory J Chaitin, Marc A Auslander, Ashok K Chandra, John Cocke, Martin E Hopkins, and Peter W Markstein. Register allocation via coloring. Computer languages, 6(1):47-57, 1981.
Chase Cummins and Richard Veras. Reinforcement learning for graph coloring: Understanding the power and limits of non-label invariant representations, arXiv preprint arX/v:2401.12470, 2024.
Pádraig Cunningham, Matthieu Cord, and Sarah Jane Delany. Supervised learning. In Machine learning techniques for maltimedia: case studies on organization and retrieval, pp. 21-49. Springer, 2008
Dibyendu Das, Shahid Asghar Ahmad, and Kumar Venkataramanan. Deep learning-based hybrid graph- coloring algorithm for register allocation. arXiv preprint arXic:1912.03700 2019.
Irit Dinur and Samuel Safra. On the hardness of approximating minimum vertex cover. Annals of mathe- matics, pp. 439-485, 2005.
Hao Dong, Hao Dong, Zihan Ding, Shanghang Zhang, and T Chang. Deep Reinforcement Learning. Springer, 2020.
Paul Erdos, Alfréd Rényi, et al. On the evolution of random graphs. Publ. math. inst, hung. acad, sci, 5(1): 17-60, 1960.
Damien Ernst and Arthur Louette. Introduction to reinforcement learning. Feuerriegel, S., Hartmann, J., Janiesch, C., and Zschech, P. P.(2024). Generative ai. Business & Information Systems Engineering, 66(1): 111-126, 2024.
Thomas A Feo, Mauricio GC Resende, and Stuart H Smith. A greedy randomized adaptive search procedure for maximum independent set. Operations Research, 42(5):8600 878, 1994.
Michael R Garey and David Johnson. The complexity of near-optimal graph coloring. Journal of the ACM (JACM), 23(1):43 49, 1976.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Michael R. Garey and David S. Johnson. Computers and Intractability; A Guide to the Theory of NP- Completeness. W. H. Freeman & Co., USA, 1990. ISBN 0716710455.
Michael Garey, David Johnson, and Larry Suockmeyer. Some simplified ng-complete problems. In Proceedings of the sirth annual ACM symposium on Theory of computing, pp. 47-63, 1974.
Lukas Gianinazzi, Maximilian Fries, Nikoli Dryden, Tal Ben-Num, Maciej Besta, and Torsten Hoefler. Learn- ing combinatorial node labeling algorithms. arXiv preprint arXiv:2106. 03594, 2021.
Fred Glover. Tabu scarch-part ORSA Journal on computing, 133):190 206, 1989.
Jean-Louis Goffin and Jean-Philippe Vial. Convex nondiffercntiable optimization: A survey focused on the analytic center cutting plane method. Optimization methods and software, 17(5):805-867, 2002.
Jack E Graver. On the foundations of lincar and integer linear programming i. Mathematical Programming. 9(1):207-22) 226, 1975.
Gurobi Optimization. Gurobi optimizer reference manual, 2024. URL 12.0, Available at
com
Version
com
András Gyárfás and Jenö Lehel. On-line and first fit colorings of graphs. Journal of Graph theory, 12(2): 217-227, 1988.
William K Hale. Frequency assignment: Theory and applications. Proceedings of the IEEE, 68(12):1497 1514, 1980.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Advances in neural information processing systems, 30, 2017.
Alain Hertz and D de Werra. Using tabu search techniques for graph coloring. Computing, 39(4):345-33 351, 1987.
Jeroen Hoebeke, Ingrid Moerman, Bart Dhoedt, and Piet Demeester. An overview of mobile ad hoc networks: applications and challenges. Journal- Communications Network, 3(3):60-66, 2004.
Jiayi Huang, Mostofa Patwary, and Gregory Diamos. Coloring big graphs with alphagozero. arXiv preprint 3rXiv:1902.10100 2019.
IBM. Ibm ilog cplex optimization studio, 2024.
URL httts://www.ibb.comuu
ilog- wwwpleee-ppiiiitttioonsssuauaaae
Ali Zeeshan Ijaz, Raja Hashim Ali, Nisar Ali, Talha Laique, and Talha Ali Khan. Solving graph coloring problem via graph neural network (gnn). In 2022 17th International Conference on Emerging Technologies (ICET), pp. 178-183. IEFE, 2022.
Michael Jordan and Tom M Mitchell. Machine learning: Trends, perspectives, and prospects. Science, 349 (6245):255-20) 260, 2015.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 2014.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent. Transportation Systems, 23(6):4909-499 2021.
Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. Numba: A llvmd python jit compiler. In Pro- ceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC, pp. 1-6, 2015.
M. Lapan. Deep Reinforcement Learning Hands-On On: Apply modern RL methods to practical problems of chathots, robotics, discrete optimization, web automation, and more. Packt Publishing, 2020. ISBN 838820004 URL httpps//books.. gow.le...oggge...oooo.ooo
15

---

Published in Transactions on Machine Learning Research (09/2025)
Yin Tat Lee, Aaron Sidford, and Sam Chiu-wai Wong. A faster cutting plane method and its implications for combinatorial and convex optimization. In 2015 IEEE 56th Annual Symposium on Foundalions of Computer Science, pp. 1049 1065. IEEE, 2015.
Henrique Lemos, Marcelo Prates, Pedro Avelar, and Luis Lamb. Graph colouring meets deep learning: Effective graph neural network models for combinatorial problems. In 2019 IEEE 31st International Conference on Tools with Artificial Intelligence (ICTAI), pp. 879- IEEE, 2019.
Jure Leskovee and Andrej Krevl. SNAP Datascts: Stanford large network dataset collection. http://ssaa snap. leaafoordeddd/datteee June 2014.
Jinkun Lin, Shaowei Cai, Chuan Luo, and Kaile Su. reduction based method for coloring very large graphs In Proceedings of the Twenty-Sirte International Joint Conference Artificial Intelligence, IJCAI-17, pp. 517-523, 2017. doi: 10.24963 (0.24963/ijcai.2011 (ifcaii221/77333 URL https::/doi..rgg/ 249g:///a..993/ccc...... /
Arno Luppold, Dominic Ochlert, and Heiko Falk. Evaluating the performance of solvers for integer- linear programming, 2018. URL httt::/tubdok.. tub tubb. hathhh/hhhannleee4/////a//3
Raymond Miller and David E. Muller. A problem of maximum consistent subsets. Technical report, IBM Research Report RC-240, JT Watson Research Center, Yorktown Heights, NY, 1960.
Eduardo Morales, Rafael Murrieta-Cid, Israel Becerra, and Marco A Esquivel- Baaiiiell-aaadaa survey on deep learning and deep reinforcement learning in robotics with a tutorial on deep reinforcement learning. Intelligent Service Robotics, 14(3))773-805 2021.
David R Morrison, Sheldon Jacobson, Jason J Sauppe, and Edward C Scwell. Branch-and-bound algo- rithms: survey of recent advances searching, branching, and pruning. Discrele Oplimization, 19: 79-102, 2016.
Christos H Papadimitriou and Kenneth Steiglitz. Combinalorial oplimization: algorithms and complexity. Courier Corporation, 1998.
Martin L Puterman. Markov decision processes. Handbooks in operations research and management science, 2:331-434, 1990.
Antonin Raffin, Ashley Hill, Adam Gleave, Anssi Kanervisto, Maximilian Ernestus, and Noah Dormann. Stable-basclines:: Reliable reinforcement learning implementations. Journal of Machine Learning Re- search, 22(268):1 8, 2021.
Ryan A. Rossi and Nesreen K. Ahmed. The network data repository with interactive graph analytics and visualization. In AAAI, 2015. URL BoLLaatttts///eeeorrrppssstttt
Pablo San Segundo. A new dsatur-based based algorithu for exact vertex coloring. Computers & Operations Research, 39(7):1724-1788 2012.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabricle Monfardini. The graph neural network model. IEEE transactions on neural nelworks, 20(1):61-80) 1800 2008
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. CoRR, fabs/1707.06344 2017. URL httt:://aariv..org////aae/
wrt.:/iiiwwrrr//bae. httpg//bb//17777 063477
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-85555 2017.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second odition, 2018. URL httt:://tncoppleeetdeet net/bookkthhe-oos 2nd. html
Michael Trick. Color02/03/04 benchmark datasets. https://mat. tepper.......... cmu, edu/COLOR02/ 2002-2004.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Juan Pablo Vielma. Mixed integer linear programming formulation techniques. Siam Review, 57(1))3-57, 2015.
Bernhard Friedrich Voigt. Der handltngsreisende,, wie er sein soll und was er zu thun hat, um aufträge zu erhalten und eines glücklichen erfolgs in seinen geschäften gewiss zu sein. Commis- Voageur, Ilmena,, pp. 69-72, 1831.
Christopher JCH Walkins and Peter Dayan. Q-learning. Machine learning, 8(3):279 292, 1992.
George Walkins, Giovanni Montana, and Juergen Branke. Generating a graph colouring heuristic with deep q-learning and graph neural networks. In International Conference on Learning and Intelligent Optimization, pp. 491-505. Springer, 2023.
Duncan Watts and Steven Strogatz. Collective dynamics of 'small-world'eeeworkk. nature, 393(6684): 440 1400442, 1998.
Dominic JA Welsh and Martin B Powell. An upper bound for the chromatic number of graph and its application to timetabling problems. The Computer Journal, 10(1):85-88) 86, 1967.
Marco A Wiering and Martijn Van Otterlo. Reinforcement learning. Adaptation, learning, and optimization, 12(3):729, 2012.
A Additional Details about VCoIRL Framework
A.1 VCoIRL Architecture
Figure illustrates the architecture of VCoIRL, which consists of agent and an environment. The agent interacts with the environment by exploring different actions and stores key information, including the state, action, reward, and episode termination status, in a replay buffer. The returns and advantages (defined in A Appendix A.3) are calculated using data from the replay buffer, which are then utilized by the optimizers as feedback to minimize the total objective function defined Appendix A.3
Agent
Episode
Environment
Graph
Length
Actor
Critic
Register Function
MPZINILLEO
Initial State Observation
Graph
Roggh Heg sty
GNN
NND
Step Function
Adventages
Returns
Predicted Value
Action
Action
Tranetion (Ulstese
Action
Ietal State
Final Stale
Final State Observation,
Retums and Advantages
Reward,
Calculator
Episode Termination Status
Reward Calculato
Samplee
Replay Buffer (Data store)
Reward
Figure Architecture of VColRL
A.2 GraphSAGE Architecture
The GNN component in Figure 4 follows the GraphSAGE architocture (Hamilton et al. 2017). illustrated in Figure where the vertex feature vectors represented using solid arrows, while the flow of information is
17

---

Published in Transactions on Machine Learning Research (09/2025)
Sampling the uncolored graph
Feature vector normalization
1
Node Degree
For feature vecsors
Input graph with
Sampling and Normalization
Neighborhood Sampling
node feature vectors
+
Node
Sampled feature aggregation
Feature normalization
Feature concatetion
Feature Aggregation and Concatenation
Note: Sampling, aggregation, and concatenation
pperatins gats exscuded the
Activation Function
simultanoouslyy order get
W
+
B
Concatensted fexture matrix (aam_odde, [eIlnnu_faaturee
Weight matric [2"lnputffeatured features,, output feotures)
Bax matris (numt_ades,, feotures
Layer output mathic (sum_nodes, features)
Loornable
Transformation
Figure 5: GrophSAGE architecture
depicted with regular arrows. It has three modules: Sampling and Normalization, Feature Aggregation and Concalenation, and Transformation.
In the Sampling and Normalization module, first, the subgraph containing uncolored vertices sampled, and then the vertex features are normalized by multiplying each feature vector by the inverse square root of the vertices' degree, thereby stabilizing the learning process. After the normalization, each vertex samples
18

---

Published in Transactions on Machine Learning Research (09/2025)
its entire neighborhood to aggregate the vertex features of neighbors to be able to capture the information about its neighborhood. GraphSAGE) typically allows for sampling subset of neighboring vertices; however, in our approach, we sample the entire neighborhood. This ensures that all relevant vertices are considered for efficient processing, which is essential for accurate feature aggregation.
In the Feature Aggregalion and Conculenation module, the features of the sampled vertices are aggregated and normalized again. The aggregated features are then concatenated with the original vertex features, resulting in feature vector that is twice the length of the imput vector. This helps to retain the information about the vertex and its neighborhood, providing comprehensive representation for further processing.
In the Transformation module, the concatenated and aggregated features (represented as a matrix with the number of rows equal to the number of vertices in the graph) are multiplied by a weight matrix, followed by the addition of a bias matrix. Both these matrices are learnable components of our model. An activation function then applicd to produce the final output, enabling the model to capture complex relationships in the data. The final output feature vectors then become the input for the next layer, allowing the model to propagate information through multiple layers.
A.3 Proximal Policy Optimization with Entropy Regularization
The optimizer component in Figure uses the Proximal Policy Optimization algorithm (PPO, (Schulman et al.] 2017)) to train the agent to solve the VCP problem. PPO is an actor-criti--based reinforcement learning algorithm that employs two parameterized networks: the policy network <(a (;; and the value network (s);w), where 0 and w are the parameters of the function estimators. In our case, these are Graph Neural Networks with GraphSAGE (Hamilton et al. 2017 architecture as shown in Figure
The policy network x(a s;; 0) provides the probabilities selecting an action at, given a state 8t at time t. The action is determined by sampling from this probability distribution.
On the other hand, the value network Vss;w) estimates the expected return Er | sr] from the current state St, where G. defined as the discounted sum of future rewards:
G. =
rrtk
1=1
Here, Ft-k denotes the reward received time k, and y is the discount factor that balances the importance of immediate versus future rewards.
Additionally, PPO uses the advantage function to improve the policy. The advantage function Å, is defined as the difference between the actual return from the episode and the estimated value of the state predicted by the critic network:
G.-
The objective function of PPO involves maximizing a surrogate loss function to improve the policy while ensuring stability. The objective function can be expressed as:
Lactor (One)) = E, min (r:(0)Ar, clip (r.(0),1-1)) -,,0,11,,,++)] e))+)))))
where (0) xraa 1e:0ame is the probability ratio of the new policy (the policy being optimized the (eiPaa)
gradient step) to the old episode-generating policy, and e is hyperparameter that defines the clipping range to prevent. large policy updates.
In addition to the policy loss, we have a critic loss and an entropy regularization term to guide the training process. The critic loss is responsible for training the value network V(s:w) by minimizing the difference between the predicted value from the value network of the new policy (the policy being optimized in the
19

---

Published in Transactions on Machine Learning Research (09/2025)
gradient step) and the actual return from the episode. This is done using a squared error loss between the value estimate and the discounted return Gt. The critic loss function is given by:
Ceritie (Winew) = Es (V(ss: Gi)
The entropy regularization term is added to encourage exploration and prevent premature convergence to suboptimal policies. The entropy of the policy x(a; 8r; nnw)) measures the uncertainly the action selection and is defined as:
H(( st; (new)) = Σ T(a st; Onew) log ( 825 knww)
For using an optimizer that works by minimizing objective function, we must adjust the signs of the aclor loss and the entropy term accordingly. The actor loss should be negated because want to maximize the policy objective, while the critic loss remains as it is, as we want to minimize the value estimation error. The entropy term is subtracted to maximize exploration by encouraging higher entropy.
Thus, the total minimization objective function for PPO, combining the actor loss, critic loss, and entropy regularization, is given by:
Leotal (6 Onewy Wnew) Lactor (actto((@cc))) (fnew) Ci (erite( C2 (2(((((( st; nnew))
where (1 C2 E R are hyperparameters that control the balance between actor loss, critic loss, and entropy regularization.
In summary, the actor learns to adjust the policy to increase the probability of selecting actions in state for which the advantage is maximized, thereby promoting actions that yicld higher returns. The critic loss helps the agent predict the returns from states, while the entropy regularization encourages the exploration of the action space, leading to more robust policies.
Implementation of Baselines
B.1 FF
First Fit (FF) is an on-line coloring algorithm (Gyárfás & Lchel 1988) in which vertices are processod sequentially, without looking ahead recoloring previously assigned vertices. It assigns each vertex the smallest available color (considering the colors named integers) such that no conflicts arise.
B.2 TabucolMin
This algorithm an iterative metaheuristic designed to minimize the number colors required to prop- erly color a graph, leveraging the Tabucol algorithm as a subroutine. Hertz & Werra (1987) contains the pseudocode and details about the Tabucol algorithm. TabucolMin begins with an initial color count k; and attempts to color the graph. If the graph algorithm reduces the color count by and retries. Conversely, if the color count cannot be reduced further and no solution has yet been found, the algorithm increases the color count by h to expand the scarch space and retries, whereas if the color count cannot be reduced further and solution has been found, the process stops. Crucially, the reported runtime accounts only for the search space where a solution was successfully found, excluding any computational effort expended unsuccessful attempts, such as the initial trials with fewer colors. This ensures that the timing reflects only the effective search for the minimal coloring solution. For our experiments, k = 15.
be successfully colored with the current number of colors, the
20

---

Published in Transactions on Machine Learning Research (09/2025)
B.3 VCoIMIS
We develop VCoIMIS based on the Maximum Independent Set (MIS) reduction approach. Given graph, we first compute its MIS and assign color 1 lo the verlices in this set. Nexi, we consider the subgraph of the remaining uncolored vertices, compute its MIS, and assign color 2 to this set. This process repeats until vertices are colored. To compute the MIS, we use the RI-based model proposed by AAhn et al. (2020) trained on the same dataset as our model with the hyperparamcters recommended by the authors. This model takes graph as input and outputs its maximum independent set.
B.4 Gurobi 12 Solver
Given a color set C and graph G = (V, we use the Gurobi 12 Optimizer (Gurobi Optimization 2024 to solve the following integer linear programming model for the verlex coloring problem.
Minimize
2e
subjected to: Tv. + Fa.e S Zo.
((, VeEC
101
Σ To, =o,==1, AVEA
Tv.c E {0,1}, ic E (0,1), WE VeEC
Here, Tn.e binary variable that indicates whether vertex is assigned color e, and 2e is binary variable that indicates whether color e is used in the solution.
The algorithm starts with initial color set size of k and iteratively increases the size by le if the solver fails to find a feasible solution. The reported runtime includes only the computational effort spent in the search space where a solution is successfully found, excluding any time spent on earlier, unsuccessful attempts with smaller color sets. For our experiments, 15.
B.5 FastColor
FastColor (Lin et al. 2017) is a reduction-basse method for solving the VCP. It is based on the concept of a bounded independent set. The algorithm recursively removes an I-degree bounded independent set from the graph, where is a lower bound on the chromatic number. key property of this approach is that any optimal coloring of the reduced graph can be extended to an optimal coloring of the original graph by coloring the removed vertices using the construction rule proposed by the authors. We implemented the algorithm in Python based on the pscudocode provided in the paper. While the results reported in the original paper are based on a C++ implementation, we chose to use Python to ensure consistency and fair comparison with our other baselines and models, which are all implemented in Python. Our implementation may not exactly match that of the original paper, as we follow the provided pseudocode due to the unavailability of the original source code.
B.6 ReLCol
ReLCol (Watkins et al. 2023 is an RL model that leverages Deep Q-Networks (DQN) and Graph Neural Networks (GNN) solve the VCP. The model selects the next vertex be colored and searches for the smallest valid color that can be assigned while ensuring proper coloring. This process continues iteratively until all vertices colored.
21

---

Published in Transactions on Machine Learning Research (09/2025)
C Additional Experiments
C.1 Hyperparameter Tuning
The hyperparameters Discount factor for PPO and Clip value for PPO set to 1 and 0.2, respectively, The Discount factor for PPO is set to to ensure that rewards from all steps are equally weighted as required for the VCP, while the Clip value for PPO is set to 0.2, as it is commonly used in several implementations. including Stable Baselines3's (Raffin et al. 2021 PPO. We use the Optuma framework (Akiba et al., 2019) with its Tree-structured Parzen Estimator (TPE) sampler and Median Pruner to tune the remaining hyper- parameters. The Optuna experiment iteratively samples hyperparamcters using the sampler and trains the model. To save time, unpromising trials are pruned by the pruner before completing all epochs. Figure 0 illustrates example of the Optuna experiment, where the number of epochs is set to 70.
Average returns
Epoch
Figure Optuna eaperiment. Each curve represents a trial in the experiment.
We begin the tuning process by first finalizing the Hidden dimension and Number of hidden layers. We explore the combinations of Hidden dimension = (64, 128, 256, 512] and Number of hidden layers = {3, 33,4,7 4,5) 5] sampled by the TPE sampler while setting the other hyperparameters to the values suggested by Ahn lt al. (2020) for the MIS problem. Compared the final selected configuration of Hidden dimension and Number of hidden layers, all other combinations either give equivalent worse performance in terms of maximizing the reward. Balancing performance and complexity, finalize the architecture with Hidden dimension of 128 and Number of hidden layers set to 4. After finalizing the model architecture, we run the tuning experiment again with the remaining hyperparameters. We find that most hyperparameters have minimal impact on performance, except for Learning rate of optimizer, Critic loss coefficient, and Entropy regularization coefficient. Optuna's 's hyperparameter importance analysis tool highlights these three as having the most significant effect on performance. Consequently, we fix the remaining hyperparameters to the values suggested by Ahn al. (2020), except for the Gradient norm, which set to to prevent the gradient step from becoming too small. To arrive at the final hyperparameter setting present in Table [ we conduct the final round of optimization involving the key hyperparameters Learning rate of optimizer, Critic loss coefficient, and Entropy regularization coefficient.
The values and ranges chosen for the Optuna experiments are as follows: Critic loss coefficient e [0.1, 0.9]. Entropy regularization coefficient [0, 0.5), Gradient norm {0.5, 1,3,5), Learning rate of optimizer e [10-5, 10 10. Hidden dimension {64, 128. 256, 512), Number of hidden layers {2, 4,5), Number of gradient steps per update E (2, 4), Batch size for gradient step E {4,8, 16), Rollout batch size (16,32 64).
22

---

Published in Transactions on Machine Learning Research (09/2025)
Table 8: Tuning the cutoff timer for FostColor (F), The namerical values in the column headers represent multiples of VColRL's rantime.
Graph Type
VCoIRL
F (1x)
F (5x)
F (10x)
F (20x)
F (10x)
(F 80x)
F (160x)
ash608GPIA
5, 0.04
7, 0.04
5, 0.20
ash958GPIA
5, 0.06
7, 0.06
5, 0.34
le450 5a
6, 0.23
9, 0.23
8, 1.15
8, 2.32
7, 4.63
T, 9.28
7, 18.43
6, 32.41
le450 5b
6, 0.08
9, 0.08
9, 0.42
9. 0.82
1.62
9, 3.22
7, 6.41
6, 14.40
le450 5c
5, 0.04
10, 0.04
6, 0.20
6, 0.43
6, 0.80
1.60
5, 3.50
le450 5d
5, 0.08
11, 0.08
7, 0.40
6, 0.85
6, 1.58
6, 3.32
5, 6.78
C.2 Cutoff Timer Tuning of Baselines
this section, we focus on tuning the cutoff time for the baseline algorithms. Among them, FastColor and Gurobi allow setting cutoff time.
We begin by rumning Colo with cutoff time equal VColRL's runtime and find that it underperforms 10% of the benchmark graphs. On these graphs, VColRL uses average of 5.33 colors, while FastColor uses 8.83 colors, which is approximately 1.65x more, or 66% additional colors. To determine how much time FastColor requires to match VCoIRL's performance, we gradually increase its cutoff time on each graph using multiples of VCoIRL's runtime (c.g., 5x, 10x, and so on) until achieves comparable color usage. We observe that, depending on the graph, FastColor requires between 5x and 160x more time than VCoIRL. This highlights the significant runtime advantage of our approach.
Specific results provided in Table 8 the graphs ash608GPIA, ash958GPI, 1e450_ 5c, 1e450_s 5d, 1e450_55 _b, and 1e450_6 5a lake 0.20, 0.34, 3.50, 6.78, 14.40, and 32.41 seconds respectively to match VColRL's color usage. Since FastColor is search-based algorithm, these times may vary slightly across runs. Based on these observations, we could select 30 seconds as a fixed cutoff time for FastColor if the goal is to match VCoIRL's performance. However, we use a 60-second cutoff in our experiments, since FastColor achieves slightly better coloring on some graphs at this setting. Notably, this 60-second cutoff is consistent with the original FastColor paper. Even with 30-second cutoff, VColRL remains faster.
For Gurobi, when the cutoff time is equal to VCoIRL's runtime, it underperforms approximately 95% of the benchmark graphs. On these graphs, VCoIRL uses average of 5.53 colors, while Gurobi uses 14.70, which is about 2.65x more, or 165% additional colors. Even with a 60-second cutoff, Gurobi still underperforns on 31% of the graphs, using 12.27 colors on average compared to VColRL's 5.72, which is approximately 2.12x more, or 114% additional colors. To evaluate Gurobi's best-case performance, we set high cutoff time of 900 seconds.
Table 9: Comporison between the VColRL model trained on the ER graph dataset and a min of the ER, BA, and WS datasets.
Graph Type
Vertex Range
Trained on ER Dataset
Trained on Mix Dataset
ER
50-100
5.592, 0.36
5.754, 0.22
ER
100-150
6. 6.760, 0.65
6.918, 0.37
ER
150-200
7.528, 0.80
7.526, 0.50
BA
50-100
168,, 0.46
6.136, 0.28
BA
100-150
6,686, 0.51
6.534, 0.35
BA
150-200
6.898, 0.70
6.888, 0.43
WS
50-100
6.996, 0.54
7.014, 0.29
WS
100-150
7.008, 0.70
7.024, 0.38
WS
150-200
7.014, 0.82
7. 7.082, 0.49
23

---

Published in Transactions on Machine Learning Research (09/2025)
Average number of colors userd
HOM HOM
HDM
100
1.-
90
Average
80
10
70
HD0M
HDM
HDM
HDM
50
100
150
200
250
300
50
100
150
200
2.50
300
0
50
100
150
200
250
300
Epochs
Eoochs
Epochss
(a) HDM Returns
(b) HDM Colors used
(c) HDM Groph satisfaction
Average number of colors useed
HDC 1
HDC 4
HDC 3
90
salisfaccion
1.2
Averagee
2.0
70
8-
HDC
HDC
5
HDC
IIDC 5
HDC
50
100
150
200
250
300
50
100
150
200
250
300
50
100
150
200
250
300
Epochs
Eoochs
(d) HDC Returns
(e) HDC Colors used
(f) HDC Graph satisfaction
1.6
Avstage number of colors usedd
HVM
HWM
HMM
HWM
1.4
1V/
90
12
salfaction
1.2
80
Averagge
1.0
70
0.0
HWM
HWM
HWM
HWM
8-
60
HWM
HWM
HWM
0.6
150
200
250
300
100
200
Epochs
Eoochs
Epochs
(g) HWM Returns
(h) HWM Colors used
(i) HWM Graph satisfaction
1.6
Average number of colore useed
HWC
HWC
HWC
1.4
salifaciion
1.2
80
Average
10
70
HWC 1
HWC
60
RWC
HWC 4
#WC
HWC
WWC
150 Epochs
Epochs
Epochs
(i) HWC Returns
(k) HWC Colors used
HWC Craph satisfaction
Figure 7: Performance of VColRL different configurations of hard rollback.
C.3 Training with Mixed Dataset for Testing Generalizability
In the experiments conducted lo evaluate VCoIRL on benchmarks and synthetic graphs in Section 4.2 we used a model trained exclusively on ER graphs with 50-100 vertices and observed that it performs well across wide range of graph types and sizes, despite not being explicitly trained on those distributions. To further investigate the impact of training data diversity, we train new model on mixed dataset comprising ER, BA, and WS graphs. We evaluate this model the same set of test graphs used in Table 0 and compare
24

---

Published in Transactions on Machine Learning Research (09/2025)
1.6
Avarage number of colors userd
SPM
SDM 4
1.5
144
5DM
SOM
SDM
90
1.4
salisaccion
90
Neerage
1.1
SOM
50M
0
60
50K
SDM
5DM
1.0
SDM
SDM
50
100
150
200
250
300
50
100
150
200
2.50
300
50
100
150
200
250
300
Epochs
Eoochs
Epochss
(a) SDM Returns
(b) SDM Colors used
(c) SDM Groph satisfaction
1.6
Average number of colore used
13
100
SDC 2
SDC 5
1.5
salisfattion
60
Average
9
1.3
40
SDC
SDC
SDC
SDC SDC
SDC
SDC
20-
S0C
150
Epochs
Epochs
Epochs
(d) SDC Returns
(e) SDC Colors used
(f) SDC Graph satisfaction
Ausragee number of colors usod
SINM SIM 5
3.0
SWM 2
90
12
salifaccion
00
Averagg
1.3
70
60
SWM SWM
SWM
8
500
SWM
SWM
40
SWM
150
50
100
150
200
250
300
Epochs
Epochs
Epochs
(g) SWM Returns
(h) SWM Colors used
SWM Graph satisfaction
1.6
Average number colors used
12
SWC
SIC 4
100
SWC
SWC
1.5
31
SWC
80
10
1.4
80
1.3
40
SWI 2
SWC 5
SINC
SWC
5
1.2
SWC
100
150
200
250
300
100
150
200
300
100
150
200
250
300
Epochs
Epochs
(i) SWC Relurns
(k) SWC Colors used
(1) SWC Gruph salisfaction
Figure& 8: Performance of VColRL across different configurations of soft rollback.
its performance with that of the model trained solcly on ER graphs. The results are presented in Table Table g We observe a slight improvement on ER graphs in the 150 200 vertex range and across all BA graph ranges. However, performance on other graph types and sizes declines slightly. Overall, the two models perform comparably, and the observed fluctuations can be attributed to the stochastic nature of the VColRL framework. This experiment strengthens our generalization claim that the VCoIRL model trained solely F.R. graphs performs par with model trained more diverse dataset, demonstrating its ability
25

---

Published in Transactions on Machine Learning Research (09/2025)
to generalize effectively across variety of unseen graph distributions without requiring explicit exposure during training.
C.4 Multiple Runs of MDP Configurations
In Section 1. compared various MDP configurations and concluded that HDM outperforms all other configurations. To further verify the claim, we train each variant five times and report the performance in Figure f and Figure 8 While all variants achieve similar average returns and color usage, the HDM configuration consistently yields higher graph satisfaction and more stable policies across runs. This is especially important in the vertex coloring problem, where ensuring that all vertices are properly colored while also minimizing color usage is paramount. However, coloring all vertices takes precedence over reducing the number of colors used. As discussed in Section 3.1 our reward design reflects this by assigning greater weight to vertex satisfaction. Notably, the other variants, despite achieving comparable average returns and color usage to HDM, leave few vertices uncolored when it comes to graph satisfaction, which is an undesirable outcome. These results underscore the effectiveness of HDM in producing robust policies that better meet the core objectives of the task.
D Discussions
D.1 Unique Challenges of VCP Compared to MIS in Reinforcement Learning
We outline three key differences that make VCP more challenging for reinforcement learning compared to the MIS problem, and how our approach addresses them:
Larger Action Space and MDP Modeling Complexity: Unlike MIS, where each vertex has binary dccision (inchudo/cxclude), VCP requires assigning one of many color labels per vertex, leading to a significantly larger action space, thereby increasing the modeling complexity. This complexity necessitates careful design of the MDP. In our work, we systematically study different combinations of actions, transitions, and reward strategics to identify the configuration that enables stable training and strong performance.
2. Stronger residual dependencies in partially solved graphs: When solved with reduction- based method, MIS and VCP exhibit different behaviors. In the MIS problem, once vertex is included, its neighbors are naturally excluded, allowing the agent to operate independently on the remaining subgraph since those neighbors are no longer part of it. In contrast, in the VCP, assign- ing a color to a vertex restricts the choices of its neighbors but does not uniquely determine their coloring. As a result, the neighbors remain in the subgraph, so learning cannot rely solely on the uncolored portion; it must also incorporate information from already-colored neighbors. Our frame- work addresses this by designing vertex features that encode neighborhood color usage, enabling more informed and context-aware decision-making.
Equivalent solutions: Modeling VCP as a classification problem challenging due to label am- biguity. In MTS, each vertex receives binary label with fixed interpretation, indicating whether it belongs to the independent set or not. In contrast, in VCP, color labels are arbitrary identifiers without inherent meaning, as long as adjacent verlices receive different labels. For example, consider coloring four vertices with labels from (1,2,3,4] 4}. The assigmments [1, 3, 1] and [1, 4, 1] repre- sent the same partitioning of vertices into color classes but differ in their label values. From the perspective of an RL agent learning a discrete probability distribution, these assignments appear different and therefore should receive different penalties, enabling the agent to distinguish between them and reduce ambiguity. To this end, we introduce a maz-color reward that penalizes the use of higher-indexed colors, encouraging contiguous and consistent labeling. This mitigates ambigu- ity arising from equivalent parlitioning expressed with different labels and provides more stable learning signal.
26


Reviews:
[{"review_id": "WDJD0G8jN1", "reviewer": "Reviewer_rP69", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nS1. The paper in general is well-organised and well-written.\n\nS2. The work is technically correct and thoroughly evaluated. The systematic exploration of the design choices, while not incredibly novel, is a substantial strength of the work.\n\nS3. The proposed method shows better results than the competing approach in the same category (i.e., reinforcement learning techniques).\n\n### Weaknesses\n\nW1. The heuristic and exact methods seem to obtain the same or better task performance on essentially all problem instances. Therefore, it is hard to argue for approaching this problem with RL given very strong algorithms already exist and such approaches require an expensive training stage. However, considering the cheap inference costs and S3 above, I believe it is worth publishing this work as another choice in the landscape of vertex coloring methods.\n\nW2. The approach is heavily tailored to a single problem and cannot be applied to others.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "I would like to preface my review by noting that I have also reviewed this paper recently, in the first half of the year. The difference between the current version and the one I have reviewed are meaningful, and include both additional results (e.g., with the FastColor baseline; more graphs; results are averaged over many seeds) and editing of the writing. My opinion of the paper, then as now, is a positive one overall. My requested changes are mostly presentational.\n\nThis paper addresses the vertex coloring problem, which asks to assign colors to vertices of a graph such that no two adjacent nodes have the same color, with the objective of minimising the number of colors used. It treats this problem in the reinforcement learning framework, formulating it as (several variants of) a Markov decision process. The proposed approach uses PPO and a graph neural network for learning vertex coloring policies through trial-and-error. The authors first study several possible choices in the design of the MDP with respect to the action space, transitions, and reward functions. The method is compared experimentally with a competing RL+GNN algorithm and several heuristic and exact baselines. The results show better performance than the RL competitor, and performance comparable / slightly worse than the non-learned baselines, but has reduced computational time compared to them.\n\nThis work contributes to the broader literature of machine learning for combinatorial optimization. It improves on a recent approach in this line of work, and presents a method that is quite performant and scalable, reusing the \"learning to defer\" approach popularised for maximum independent sets.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some of these are presentational points that I raised in my previous review but I see that they were not addressed in this submitted version.\n\nC1. It is unclear to me, based on the explanation in the text, why both $\\*$ and $0$ symbols are needed to indicate that a vertex is not colored. It appears that all the way through the MDP execution only $\\*$ is used.\n\nC2. Results in Figure 3 are hard to distinguish, they should be complemented with results in a table format.\n\nC3. Algorithms for finding optimal solutions already exist. Therefore, an alternative to the expensive model-free RL learning process would be to use imitation learning to mimic these algorithms, either on its own or as a \"warm-start\" for further RL. Doing this would likely lead to a more stable policy performance-wise that can be trained quicker. This should be discussed as a possible direction for future work. Furthermore, the method effectively performs greedy search at inference time. Integrating the policy as priors for a more advanced search process should also lead to improvements over the base method.", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "8W2IiUlnK1", "reviewer": "Reviewer_uS46", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n* The paper introduces a new MDP formulation tailored to the vertex coloring problem, which is clearly and carefully described.\n* The effectiveness of the proposed approach is supported by large-scale and reasonably comprehensive numerical experiments.\n* The experimental evaluation includes not only various datasets but also tests on graph instances that differ from the training data, demonstrating some degree of generalization.\n\n**Weaknesses:**\n\n* The motivation and positioning of the research could be elaborated further. In particular, it would be helpful to provide concrete scenarios in which solving the vertex coloring problem efficiently is of practical importance. Furthermore, compared to other combinatorial optimization problems where reinforcement learning has been applied (e.g., TSP, independent set), it would be beneficial to structure the discussion around the unique challenges posed by vertex coloring and how the proposed approach addresses them.\n* There are some concerns regarding the validity of the comparisons in the experimental section. For example, solvers like Gurobi and FastColor allow tuning of parameters such as the CutOff time. It is conceivable that these solvers may often find high-quality approximate solutions well before the imposed time limit, and thus their performance might appear significantly better if the CutOff time were reduced. In Table 2, for instance, FastColor seems to find solutions of comparable quality to VColRL in almost all cases by the CutOff time; one might wonder whether FastColor would achieve similar quality even with the same CutOff as VColRL.\n  Although the paper mentions that the CutOff time was set in accordance with prior work, the proposed method appears to involve some hyperparameter tuning. For a fairer comparison, it might be preferable to also allow similar tuning for the baseline methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper investigates a reinforcement learning-based approach to the vertex coloring problem on graphs. To enable the application of reinforcement learning, the authors formulate the vertex coloring task as a Markov Decision Process (MDP), and empirically evaluate and compare their proposed method. Specifically, the paper compares different ways of formulating the MDP, and benchmarks the RL-based approach against traditional methods based on Mixed Integer Programming (MIP) and heuristics. Based on the experimental results, the authors claim that the proposed method outperforms traditional approaches in terms of computation time.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would appreciate a response to the concerns raised above. For the numerical experiments, I am not necessarily requesting additional experiments; rather, a discussion of the authors’ perspectives and insights gained so far would be sufficient.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Nz7qrbcQh3", "reviewer": "Reviewer_qwrG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths \n- The MDP setup is intuitive, and the three design choices on which the authors focus are meaningful and interesting. \n- That the method can be trained entirely on ER synthetic graphs and then applied broadly is practically great and also interesting.\n- The gains in speed over alternatives are significant.\n- The writing is clear and the narrative is focused.\n\n## Weaknesses\n- Experimental results have weak spots -- see below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission proposes an MDP formulation of vertex coloring friendly to RL, based on repeatedly assigning colors in bulk and then rolling back conflicts to be assigned anew in the next step.  The authors explore three binary choices in the MDP configuration -- whether to roll back conflicts entirely or only the most recent assignments, whether it is possible to assign “undecided” to a node as a means of strategic postponement, and which of two ways to count the colors used when computing the reward -- and compare all eight variants to select the most performant combination. They then compare their model trained exclusively on Erdos-Renyi graphs to several baselines on several datasets, and achieve competitive performance with significant speedups.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- The experimental support for the MDP design decisions is weak -- only one training run per configuration.  As the exploration of several alternatives in the formulation is a primary result for the paper, firmly grounding the superiority of HDM with more runs (which should not be too expensive) is critical to acceptance.\n\n- The sequential nature of the MDP comes about by doing a bulk assignment and then rolling back.  It would be helpful to see a variant that is closer to greedy (and closer to how a human would approach VCP), where a single undecided node is randomly selected to be updated at each step, with and without deferral.  This would isolate the effect of assigning in bulk vs sequentially, without requiring changes to the policy architecture or training objective.\n\n- Table 5 is great and highlights one of the strengths of the paper -- that training on Erdos-Renyi graphs alone is able to generalize decently to graphs with other connectivity structures.  It also raises a clear question -- why not vary the training dataset with mixes of these different synthetic graphs, and check if performance increases with the variety?\n\n- I don’t see the number of GraphSAGE message-passing steps reported, and this is important in determining the context that each node has.  It would be helpful to include this in the experimental specifics.\n\nNits:\nSpell out ER when first introducing (pg 6)", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)