Regret-Guided Search Control for Efficient Learning in AlphaZero
Authors: Yun-Jui Tsai
Wei-Yu Chen
Yan-Ru Ju
Yu-Hung Chang
Ti-Rong Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Search Control
Reinforcement Learning
Regret Prioritization
Monte Carlo Tree Search
AlphaZero
Primary area: reinforcement learning
TL;DR: We propose regret-guided search control, extending AlphaZero with regret-guided restarts that yield more efficient and robust learning in board games.

Abstract: Reinforcement learning (RL) agents achieve remarkable performance but remain far less learning-efficient than humans. While RL agents require extensive self-play games to extract useful signals, humans often need only a few games, improving rapidly by repeatedly revisiting states where mistakes occurred. This idea, known as *search control*, aims to restart from valuable states rather than always from the initial state. In AlphaZero, prior work Go-Exploit applies this idea by sampling past states from self-play or search trees, but it treats all states equally, regardless of their learning potential. We propose *Regret-Guided Search Control* (RGSC), which extends AlphaZero with a regret network that learns to identify high-regret states, where the agent's evaluation diverges most from the actual outcome. These states are collected from both self-play trajectories and MCTS nodes, stored in a prioritized regret buffer, and reused as new starting positions. Across 9x9 Go, 10x10 Othello, and 11x11 Hex, RGSC outperforms AlphaZero and Go-Exploit by an average of 77 and 89 Elo, respectively. When training on a well-trained 9x9 Go model, RGSC further improves the win rate against KataGo from 69.3% to 78.2%, while both baselines show no improvement. These results demonstrate that RGSC provides an effective mechanism for search control, improving both efficiency and robustness of AlphaZero training. Our code is available at https://rlg.iis.sinica.edu.tw/papers/rgsc.

Published as a conference paper at ICLR 2026
REGRET-GUIDED SEARCH CONTROL FOR EFFICIENT LEARNING IN ALPHAZERO
Yun-Jui Ts Wei-Yu Chen Yan-Ru Ju², Yu-Hung Chang", Ti-Rong 2u
'Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan institute of Information Science, Academia Sinica, Taiwan Coolleee of Computing, Georgia Institute of Technology
b08202011. csscc12222 (eai12@@ycu.ccem edu. tw, wchen784Bgatech edu {yanru, yuhung, tirongwuj@iis sinica ew
ABSTRACT
Reinforcement learning (RL) agents achieve remarkable performance but remain far less learning-effice efffciiet than humans. While RL agents require extensive self- play games extract useful signals, humans often need only few games, im- proving rapidly by repeatedly revisiting states where mistakes occurred. This idea, known as search control, aims to restart from valuable states rather than always from the initial state. In AlphaZero, prior work Go-Exploit applies this idea by sampling past states from self-play or search trees, but treats all states equally, regardless of their learning potential. We propose Regret-Guidee Search Control (RGSC), which extends AlphaZero with a regret network that learns identify high-regret states, where the agent's evaluation diverges most from the actual outcome. These states are collected from both self-play trajectories and MCTS nodes, stored in prioritized regret buffer, and reused as new starting positions. Across 9x9 Go, 10x10 Othello, and 1 1xx11 Hex, RGSC outperforms AlphaZero and Go-Exploit by an average of 77 and 89 Elo, respectively. When training on a well-trained 9x9 Go model, RGSC further improves the win rate against KataGo from 69.3% to 78.2%, while both baselines show improve- ment. These results demonstrate that RGSC provides an effective mechanism for search control, improving both efficiency and robustness of AlphaZero training. Our code is available at https://rlg. iis wwniia.edu.t//pppee...
INTRODUCTION
Reinforcement learning (RL) is the process of training agent through interaction with the en- vironment and optimizing its behavior based on rewards. The foundations of RL were originally inspired by human learning, where humans acquire new knowledge through trial-and-error experi- ences. However, despite this conceptual similarity, current RL approaches remain far less efficient than human learning (Tsividis al. 2021 lii & Sadigh (023). Consider the case of mastering the game of Go. An RI. agent such as Alpha/ero (Silver et al. 2017 2018) requires millions of self- play games to reach superhuman performance. In contrast, professional human players can achieve comparable strength after far fewer games.
One key difference lies in how learning progresses. As illustrated in Figure 1 humans do not rely on playing massive numbers of games from the beginning. Instead, they repeatedly review the critical positions where mistakes occurred and refine their understanding until those weaknesses are corrected. AlphaZero, in contrast, always restarts from the empty board and updates all positions uniformly based on the obtained outcome, which substantially increases the number of episodes required to master game.
CCoresponniig author.

---

Published as a conference paper at ICLR 2026
(a) Human learning.
(b) RL Agent (AlphaZero) learning.
Figure 1: Humans focus correcting mistakes, whereas RL always starts from the initial state.
To bridge this gap, recent studies have investigated restarting strategies to improve the efficiency of RL. This idea, originating from Sutton & Barto (2018), was formalized as the concept of search control, which refers to selecting critical starting states for the simulated experiences. Building on this principle, several works have been proposed for constructing restart distributions in RL, such as sampling from the past trajectories (Tavakoli et al. 2020), starting from states closer to the goal (Flo- rensa et al., 2017), or leveraging expert demonstrations (Uchendu et al. 2023). Go-Exploit (Trudeau & Bowling 2023) further extends this idea to the AlphaZero framework by maintaining buffer of states from play trajectories or search nodes and uniformly sampling them as new starting po- sitions. Collectively, these approaches demonstrate that choosing starting states, rather than always restarting from the initial state, can significantly accelerate RL learning. However, a key limitation of Go-Exploit is that considers all states equally. In practice, not all states contribute equally to learning progress. Many states are already mastered, while only a small subset of states are actually critical for improvement. This phenomenon becomes exacerbated in the later stages of training, as the agent's 's understanding of the game improves and mistakes become increasingly rare. This motivates the need identify and prioritize the most informative states for search control.
To address this challenge, we propose Regret-Guided Search Control (RGSC), framework that extends AlphaZero by identifying and revisiting high-regret states. Specifically, RGSC leverages regret network to detect states where the agent's evaluation diverges most from the game out- come. Since most states have near-zero regret, making direct learning of regret values challenging, design ranking-baa based objective that guides the network to distinguish the most informative states. These states are then stored in a prioritized regret buffer. By repeatedly restarting from these states, the agent focus correcting its most critical mistakes, thereby mimicking human learning and achieving more efficient training. Experimental results show that RGSC outperforms both AlphaZero and Go-Exploit across three board games, including 9x9 Go, 10x10 Othello, and 11x11 Hex, achieving an average improvement of 77 Elo over AlphaZero and 89 Elo over Go- Exploit. Furthermore, when continuing training from strong, nearly converged model in 9x9 Go for 40 iterations, RGSC still improves the win rate from 69.3% to 78.2%, whereas both AlphaZero and Go-Exploit show no improvement. Moreover, additional analysis demonstrates that RGSC suc- cessfully identifies high-regret states and systematically reduces their regret during training. In sum- mary, RGSC provides an effective mechanism for search control in AlphaZero. Our results highlight regret-guided search control as promising direction for improving the efficiency and robustness of reinforcement learning.
2
BACKGROUND
2.1 SEARCH CONTROL IN REINFORCEMENT LEARNING
The concept of search control (Sutton & Barto 2018) was first introduced in the Dyna general framework (Sutton, 1991). which integrates real experience with model-generated simulated experi- ence. this setting, simulated experience is generated through search control, which determines the starting states and actions for rollouts, rather than always beginning from fixed initial state. This allows planning to focus computation on states that provide more information to accelerate learning.
Several subsequent works have adopted the principle of search control by choosing different starting states during training. For example, Go-Explore (Ecoffet et al. 2021) addresses hard-exploration problems by maintaining a database of promising states, and periodically selecting from these states to discover high-reward trajectories. This approach allows systematic exploration of rarely visited regions and achieved state-of-the-ard results in extremely difficult environment, Montezuma's 's Re-
2

---

Published as a conference paper at ICLR 2026
venge. (Florensa et al. 2017) propose another approach by selecting starting states near the goal and gradually moving them backward, thereby constructing a curriculum in reverse to facilitate learning in sparse reward environments. tump-Start Reinforcement Learning (JSRL) (Uchendu et al. 2023) samples initial states from expert demonstration trajectories, allowing the agent to focus on mean- ingful states early in training, thereby improving sample efficiency. [aaakoli et al. (2020) provides a formal definition for exploring restart distributions by introducing a restart distribution p(s) over
states, By altering the distribution of the restart states, the learning objective is modified to
L(w) = Σ p(s) ET(o (Br(s,a) ((os,a))((ss,a)))) (=, SES
acA
where S and A denote the state and action spaces, Taa 8) is the policy, qr (8, a) is the true action- value function under т, and (8, a) is its learned approximation. The restart distribution p(s) specifies the probability of selecting state posed: (a) uniform restart, which samples from recent experiences, and (b) prioritized restart, which ranks states according to their state-value temporal-difference (TD) error.
S restart point. Two restart strategies are pro-
Moreover, search control is also widely applied at the level of task selection under curriculum-based environments. For example, several studies (Jiang et al., [021 Dennis et al., 2020 Parker- Holder iet al. 2022) adaptively sample training levels based on estimated regret, encouraging the agent to focus on levels with higher learning potential. While these methods operate at the level of task selection, our goal is to identify the most challenging states within the same game level.
2.2 SEARCH CONTROL IN ALPHAZERO
AlphaZero (iiverr et al. 2018) is reinforcement learning algorithm that can master board games such as Chess, Shogi, and Go without requiring human knowledge. The training process alternates between two phases: self-play phase and optimization phase. During the self-play phase, the agent generates games against itself by combining Monte Carlo tree search (MCTS) Browne et al. (2022) Coulom (007)) with a two-head neural network, including a policy network that outputs probability distribution over all possible actions and a value head that predicts the win rate of a given state. In the optimization phase, trajectories collected from self-play are stored a replay buffer and sampled to update the neural network, training the policy head to predict the MCTS search distribu- tion and the value head to predict the final game outcome. Although AlphaZero has demonstrated superhuman performance in board games, it requires extensive computation, especially in games with long trajectories, because every self-play game must start from the empty board. This issue is exacerbated in 19x19 Go, where single game often exceeds 250 moves, making it necessary to spend enormous computational resources to generate self-play games (e.g., roughly 1.5 million TPU-hours as reported in (Silver al. 2018)).
To alleviate this issue, several studies have incorporated search control into AlphaZero by adjusting self-play games begin from particular intermediate states. For example, KataGo (Wul 2020).
one of the current strongest open-source Go programs, proposes selecting starting states either by randomly playing several moves with the policy network by sampling and slightly modifying states from past self-play trajectories. Similar to Florensa et al. (2017). Björnsson (202)) proposes starting self-play from later stages of the game and gradually shifting the starting state toward the initial position. This approach accelerates the training process, particularly in the early phases. Recently, Trudeau & Bowling (2023) proposes Go-Exploit, which systematically investigates restart state methods within the AlphaZero algorithm. Go-Exploit maintains a buffer of states collected either from self-play trajectories (Go-Exploit Visited states Cireular archive; GEVC) or from nodes within the MCTS (Go-Exploit Search states Circular archive; GESC). For each self-play game, the agent starts from the initial state with probability x; otherwise, it uniformly samples a state from the buffer as the starting state. Go-Exploit achieves higher sample efficiency and stronger performance than AlphaZero in both Connect Four and 9x9 Go with GEVC and GESC showing similar results. However, key limitation of Go- Go-Exploit is its uniform sampling. By treating all states equally, the method fails to align with the principle of restart distribution mentioned in Eqqation I, which emphasizes prioritizing important states that provide better learning.
3

---

Published as a conference paper at ICLR 2026
3 REGRET-GUIDED SEARCH CONTROL
3.1 REGRET DEFINITION IN BOARD GAMES
We propose Regret-Guided Guided Search Control (RGSC), framework that extends AlphaZero by iden- tifying and prioritizing high-regret states as search control openings for self-play in board games, as shown in Figure 2 Unlike the original AlphaZero, as shown in Figure Tb where self-play al- ways starts from the empty board, RGSC guides self-play to begin from states with higher regret, where regret reflects positions that the current agent has not yet mastered. These states can appear either along the self-play trajectory or within the MCTS search tree. This allows the agent to fo- cus on learning and exploring unfamiliar states with greater potential for improvement. Note that Go-Exploit adopts a similar idea of restarting from previously collected states, but it samples them uniformly, which fails to capture the most informative states.
Iligh-Regret State
B
Low-Regret State
Figure 2: Overview of RGSC. The regret network selects high-regret states (red circles) from both
self-play (A) or MTCS search node (B), which serve as restart points for further self-play The newly generated trajectories can further branch out, e.g., state D originates from a restart state
To formalize this idea, we define regret as measure of the discrepancy between the agent's eval- uation and the game outcome. Given self-play trajectory with state 8p, 81, 8T and game outcome the regret of state s; is
T
R(se) = T Σ
(Vselected(s:) =)²²
(2)
where Vaefectea(s:) represents the MCTS value of the selected action at state s;. Intuitively, R(sr) measures the average discrepancy accumulated from 8, to the terminal state 8T, capturing states whose mis-evaluation has long-term impact on the outcome.
Note that R(st) is calculated only after the game finished. Morcover, the same state s may have different regret values across trajectories, since the subsequent moves and outcome vary. As training progresses and the agent's evaluations become more accurate, the regret of previously misjudged states gradually decreases. Conceptually, this resembles how human players repeatedly review their mistakes to improve.
3.2 REGRET NETWORK
Although the regret R(s,) of states on finished self-play trajectory can be directly calculated, many internal states in the MCTS search tree are not part of the actual trajectory and thus have no regret values. Nevertheless, these states may still include critical states that the agent has not yet mastered. Leveraging such states allows the agent to obtain more diverse restart states beyond the limited set of self self-play trajectories.
A naive approach is to train regref value network that, given a state, directly predicts its regret value, similar to settings learning-to-lean problems (Wang et al. 2017 [Cu et al., 2024 Gupta al. 2020). However, predicting regret value for arbitrary states in AlphaZero training is highly challenging. First, the distribution is extremely imbalanced: most states have near-zero regret, while high-regret states occur only rarely. Second, the learning target is non-stationary: high-regret states are selected for restarts, and once revisited, their regret typically decreases quickly as the agent cor- rects its mistakes. As result, predicting regret becomes extremely difficult for this naive approach.
To tackle this challenge, we propose to learn regret with a ranking- based objective. The key idea that instead of predicting precise regret values, which imbalanced and non-stationary targets,
4

---

Published as a conference paper at ICLR 2026
only need to identify which states have higher regret among all collected states. This relaxation guides the model to focus on the most informative states, ensuring that they are included in the prioritized buffer for restarting self-play.
Specifically, we incorporate a regret ranking network into the AlphaZero network. Given a state s, the regret ranking network outputs unnormalized score, Ys. where y represents the ranking score of state 8. Note that is a relative ranking score rather than the true regret value, with higher scores corresponding to states with higher regrets. For set of candidate states S, the restart distribution p(s) S) is derived as follows:
exp(1+)
( Eves exp()x))
Following Equation I
the ranking objective is to maximize
Trank
Σ p(s S)R(s),
(4)
SES
which encourages the model to assign high probability to the highest-regret states, these are the most critical for maximizing Trank and for restarting self-play.
To better optimize the regret ranking network, we apply an exponential transformation to the regret values, which preserves the ranking order. Then, the network is optimized by using a surrogate objective
Frank En(s p(s exp exp (R(+)),
(5)
#ES
and the corresponding loss function is defined as
Lrank log Trank - -log Σ S) S) exp (R(s))
(6)
SES
- log Σ (exp (log softmax(1s) + R(s)))
8E8
The derived loss can be interpreted as adding regret as additive bias to the log-softmax scores, providing smooth approximation to selecting the highest-regret states. The following is the step- by-step derivation:
Lrank log
Σ p(s (S)xexp exp (R(s))
8ES
exp(1as)
lo
exp (R(s))
Ex's es exp(Tir)
#ES
exp(Te)
log Σ
exp
log
Evess ( exp(>x.)
exp (R(s))
(10)
sES
exp(Te)
log Σ
exp
log
Eves Evesexp(>s) exp(>x:) +(((s)
(11)
8ES
log Σ
exp
log
(softmax(xx)) R(s)
(12)
SES
Although the regret ranking network can differentiate states with higher regrets, its ranking score is not bounded within the true regret value range. Therefore, to provide quantitative measurement of regret for the selected states, our regret network consists of both regret value network and regret ranking network. The regret ranking network identifies high-regret states, while the regret value network estimates their actual regret value.
3.3 PRIORITIZED REGRET BUFFER FOR SEARCH CONTROL
We describe the prioritized regret buffer (PRB). which utilizes the regret network to allow search control during AlphaZero training. For each self-play game, we first apply the regret ranking net- work to evaluate all states that appear both in the self-play trajectory and in the MCTS search trees.
5

---

Published as a conference paper at ICLR 2026
The state with the highest ranking score is then selected. If the selected state appears in the self- play trajectory, we calculate its regret value R(s) using Equation 2) if it appears only in the search tree, its regret value is estimated by the regret value network. The PRB maintains only fixed ca- pacity of K states. If the PRB is not yet full, the selected state s is added directly. Otherwise, is added only if its regret is higher than that of the lowest-regret state currently in the PRB. This ensures that the PRB consistently stores set of high-regret states for restarting.
For each self-play game, search control guides the choice of restarting state, starting from the empty board with probability 1 - X, and from a state sampled from PRB with probability A. We adopt a softmax distribution over all states in PRB when sampling to ensure high-regret states are prioritized. The probability of selecting state 8; in PRB is defined P(s;) R(ss) 4T /Σ, R(, where is the sampling temperature.
For restarting games from states in PRB, we update their regret values (s;) after replaying each game using exponential moving average (EMA):
new (s:) - (1 - a:) x Rold(s) (s:) + x R(8i):
(13)
where (s;) is the previous regret value stored in the buffer, R(s;) is the regret calculated from the newly finished self-play game, and is the EMA coefficient. This prevents regret values from decreasing abruptly and ensures that once the agent has consistently mastered this state, its regret will gradually decay, thereby reducing the probability of the state being sampled from the buffer. In summary, this design mirrors how humans repeatedly review mistakes until they are fully under- stood. We have also provided detailed algorithm for RGSC in the Appendix B
EXPERIMENTS
4.1
TOY EXAMPLE
100
1,5
Average Rewards
Average Rowards
No Search Control
0.8
0.8
Random
1 0.5
Regret
0.6
5.6
02
0.2
0K
6k
0K
4K
2X
4K
Iterations
Iterations
Iterations
Iterations
(a) n 5. Rewards
(b) MSE
(c) Rewards
(d) n
6, MSE
Figure 3: A toy example on n-level binary tree. (a) and (c) show the average rewards during the training, while (b) and (d) show the mean squared error (MSE) of the optimal Q-values during the training. The shaded area is 95% confidence interval for the mean.
We first investigate search control in a toy environment, an nl-evell sparse-reward binary tree, where each leaf node is assigned an expected reward value p E [0, 1]. The agent starts from the root and selects nodes until reaching a leaf. Upon reaching a leaf node, it receives a stochastic binary reward: I with probability p and with probability 1 1-p. p. Among all leaf nodes, exactly one node assigned to 1; thus, the objective in this environment is to discover the unique path that always guarantees a reward 1. Next, we train Q-learning on this environment with three search control methods: (a) No search control, always starting from the root; (b) Random, uniformly sampling from visited nodes; and (c) Regret, sampling nodes in proportion to their regret. For the regret, we simply use [Q(s) - Q(s, a)). where (ss) is the empirical maximum expected value estimated from all child nodes, and Q(,, a) is the current Q-value of state with action a. Figure 3 shoows the results for the 5- and 6-level binary trees. The Regret method achieves higher average rewards than both Random and No search control. These results demonstrate the importance of prioritizing states with high learning potential and show the effectiveness of the regret-guidee search control. Detailed settings of the toy environment provided in Appendix

---

Published as a conference paper at ICLR 2026
4.2 RGSC BOARD GAMES
We compare RGSC against two baseline methods: (a) AlphaZero, which is trained without search control, and (b) Go-Exploit with its GEVC variant described in (ubsecttonn 2.2] across three board games, including 9x9 Go, 10x10 Othello, and 1xx11 Hex. All methods use a 3-block residual network (He et al. 2016) and 200 MCTS simulations per move during self-play. Training runs for 300 iterations, with 160,000 states collected per iteration in 9x9 Go (due to its higher complexity) and 120,000 states in the other two games. We fix the number of training states rather than the number of self-play games per iteration to ensure fairness, since AlphaZero without search control requires more computation to generate self-play game. Although RGSC requires training two additional heads, the computational overhead is minimal. In particular, the number of blocks increases, the additional cost becomes negligible. We provide detailed training settings and analyses of the computational cost in subsection A.1 and (subsection A.2 respectively. In summary, each training requires approximately I50 NVIDIA RTX A6000 GPU hours.
1400
1000
1400-
1200-
1:000
Rating
200
k00
600
con
000
400
400
Ga-Exploit
400
ROSC
200
201 30K
33K
60
6ix
Training Steps
Traininn Steps
Training Steps
(a) 9x9 Go
(b) 10x10 Othello
(c) 11x11 Hex
Figure 4:
Playing performance of AlphaZero, Go-Exploit, and RGSC on three different board games. The shaded area is a 95% confidence interval for the mean.
Figure 4 shows the Elo curves for each method across the three board games. For each game, all models are evaluated against the 150-iteration AlphaZero model, whose Elo rating is fixed at 1000 as the reference point. When comparing the final checkpoint across all methods, RGSC consistently outperforms both baselines in all three games. In 9x9 Go, RGSC surpasses AlphaZero and Go- Exploit by 76 and 96 Elo points, respectively; in 10x 10 Othello, the improvements 70 and 50 Elo points; and in I 11x11 Hex, the differences are 84 and 122 Elo points.
Interestingly, we observe that although Go-Exploit achieves a higher Elo than AlphaZero in the early stages of training, its advantage diminishes as training converges. This phenomenon is also evident in the original Go-Exploit experiments. We hypothesize that, during early training, many states exhibit high regret since the model has much to learn, making it easy to select informative states with uniform sampling. As training progresses, however, the number of unfamiliar states decreases, thus uniform sampling becomes less effective. In contrast, RGSC continues to prioritize the remaining high-regret states, allowing focus on difficult states and maintain its advantage in the later stages of training.
Table 1: Win rate against established open-source programs on three board games.
AlphaZero
Go- Go-Exploit
RGSC
9x9 Go
45.5%
49.5%+2..0%
53.6% 43.2%
10x 10 Othello
51.7%
52.9%+3.3%%
11x11 Hex
83.6%=11.66
89.2%
91.1%
Furthermore, to assess whether the observed improvements remain consistent, we evaluate the final checkpoint by playing against established open-source programs across all three games. We select KataGo (Wu) 2020), one of the strongest open-source Go programs, for 9x9 Go; an alpha-beta search implementation in Ludii (Piette et al. 2020) for 10x10 Othello; and MoHex (Huang et al.) 2014), a MCTS-based Hex program that won Computer Olympiad championships, for 11x11 1 Hex. Detailed settings for each program are listed in suuusecction A.I Table summarizes the win rate against these opponents. The results consistent with the findings in Figure showing that RGSC consistently
7

---

Published as a conference paper at ICLR 2026
outperforms both AlphaZero and Go-Exploit Overall, these experiments demonstrate that RGSC offers a more efficient search control mechanism, resulting in higher training efficiency and stronger playing performance.
4.3 RGSC ON
MODELS
Building on the findings in subsectionn 4.2, where Go-Exploit showed early improvement but failed to yield significant progress as training converged, we now investigate whether RGSC can provide further improvements when starting from an already well-trained model. It is worth noting that mistakes become increasingly rare in such models, making it particularly challenging for the agent identify and learn from the remaining high-regret states.
To investigate this, we select large 15-block baseline model trained with the AlphaZero algorithm on 9x9 Go, which required approximately 1,060 NVIDIA RTX A6000 GPU hours and already achieves a strong playing strength. When compared against a KataGo model of the same block size, the baseline achieves a win rate of 69.3%. Similarly, we adopt AlphaZero, Go-Exploit, and RGSC using the same baseline model the initial weight to ensure fair comparison. For RGSC, since the original baseline model does not include the regret network, we add the regret network and generate additional self-play games train while keeping the policy and value networks frozen. All three methods are then continued for 40 iterations under identical settings, requiring approximately 100 NVIDIA RTX A6000 GPU hours. Additional training details are provided in subsection A.3
1350
Rating
1250
1200
1100
AlphaZero
Go-Exploit
RGSC
1000
10000
20000
30000
40000
Training Steps
Figure 5: Playing performance of AlphaZero, Go-Exploit, and RGSC when continued from well- trained models. The shaded area is 95% confidence interval for the mean.
Figure 3) shows the results of continued training. Similar to the findings in subsection 4.2 RGSC achieves the strongest performance, while Go-Exploit performs the final checkpoint, RGSC surpasses AlphaZero by 42 Elo points and Go-Exploit by 87 points. Furthermore, evaluate the final checkpoint models against KataGi. The original baseline model achieves a win rate 69.3% + 2.6%. After continued training, AlphaZero reaches 70.2% + 2.7% and Go-Exploit 69.2% + 2. 2.7%, showing a substantially higher win rate of 78.2% + 2.5%, significantly outperforming both baselines. To conclude, these results indicate that RGSC can effectively track remaining self-mistake states even in well-trained model, thereby achieving further performance improvements.
worse than AlphaZero. At
meaningful improvement. In contrast, RGSC achieves
4.4 COMPARISON BETWEEN RANKING AND REGRET RGSC
Both the regret value network and regret ranking network can be used to identify candidate states for restarting, but their effectiveness may differ. The regret value network directly estimates regret values, whereas the regret ranking network emphasizes relative ordering. In this subsection, analyze their differences in identifying high-regret states and examine the impact on search control.
We first train an RGSC variant that relies only the regret value network for both state selection and regret initialization. Figure 6 presents the training results, showing that the regret ranking network outperforms the regret value network across all three games.
Next, we examine whether the states selected by the two networks indeed correspond to high-regret states. We collect all states from self-play trajectories at each training step and evaluate them by
8

---

Published as a conference paper at ICLR 2026
1400
1500
1706-
Rating
2:00
300
600
200
600
200
200
Traininn Steps
Traininng Steps
Trainiing Steps
(a) 9x9 Go
(b) 10x10 Othello
(c) 11x11 Hex
Figure 6: Playing performance of RGSC using regret value network and regret ranking network. The shaded area a 95% confidence interval for the mean.
0.55
Regret
0.3
0.50
GeCopiil Avart Virn betwokk
0.2
0.40
10C
20K
30K
40C
50K
10K
20K
30K
50K
69K
10k
20K
30K
40K
Training Step
Training Step
Training Step
(a) 9x9 Go
(b) 10x10 Othello
(c) 11x11 Hex
Figure 7: The regret values for nodes selected by Go-Exploit, regret value and ranking network. The shaded area is a 95% confidence interval for the mean.
both networks. Since these states come directly from trajectories, their true regret values can be computed according to Equattonn We then rank the states separately with the regret value and ranking networks, and select the top 2,000 states predicted by each. The true regret values these selected states are averaged to obtain the average regret of the states identified by each network. Figure 7 shows the average regret of the states selected by each method during training. Overall, the regret ranking network consistently selects states with higher true regret than the regret value network, especially in 10x Othello. For convenience, we also include the Go-Exploit approach as baseline, where the average regret obtained by randomly sampling 2,000 states. As expected, the uniform sampling approach results in the lowest average regret among all methods. Moreover, the average regret of Go-Exploit decreases during training, especially Hex, corroborating our hypothesis that Go-Exploit becomes less effective in later stages. In contrast, the regret ranking network maintains substantially higher average regret even late training steps, indicating its ability to continually identify difficult states. In summary, these results demonstrate that the ranking objective improves the quality of selected states by prioritizing those with greater learning potential.
4.5 REGRET CHANGE IN PRIORITIZED REGRET BUFFER
This subsection examines whether the high-regret states in the PRB gradually decrease during train- ing, i.e., whether the model can actually correct its mistakes by repeatedly revisiting those states. Specifically, we record the regret of each state when first enters the PRB and compare it with its final regret before removal. Figure 8 shows the regret distributions these two points across all three games. Generally, the distributions consistently shift toward the left (lower regret values) as training progresses. This confirms that states initially associated with high regret are eventually corrected through repeated replay, resulting in reduced regret over time. In addition, by compar- ing the average regret values, we observe that the average regret decreases significantly across all games: from 0.655 to 0.296 in 9x9 Go. from 0.828 to 0.638 in 10x10 Othello, and from 0.848 to 0.657 in 1xx1 1 Hex. These results demonstrate that RGSC continuously identifies states where the agent struggles, allows them be self-corrected through repeated revisits until mastered, and then refreshes the buffer with new challenging states. More analyses on high-regret states and the game length of restart states in PRB provided in Appendix E. Fland Appendix F. respectively.

---

Published as a conference paper at ICLR 2026
City Before
1750
2400
1546
2260
Num Openteg
Num opentngs
2008
1.91 Regret
17x
8.00
0.2
0/2 1.00 1.5
2.00
2.00
0.
2.0
2.3
Regret
Regret
(a) 9x9 Go
(h) 10x10 Othello
(c) 11x11 Hex
Figure 8: Regret distributions of states in the prioritized regret buffer at entry and before removal.
DISCUSSION
This paper proposes Regret-Guided Search Control (RGSC), an extension AlphaZero that identi- fies high-regret states. By integrating a regret network and prioritized regret buffer, RGSC allows the agent to repeatedly focus on correcting its most critical mistakes, mimicking how humans learn. Experimental results show that RGSC outperforms AlphaZero and Go-Exploit, achieving an aver- age of 77 and 89 Elo, respectively. Furthermore, RGSC successfully improves the win rate against KataGo on well-trained 9x9 Go model from 69.3% to 78.2%, 2%, while both baselines show no im- provement. These results demonstrate the learning efficiency and robustness of RGSC.
Although our study focuses on board games, AlphaZero and its successor MuZero (Schrittwieser applica-
al. 2020) are general frameworks, suggesting that RGSC could be applied tions beyond games. Specifically, our preliminary experiments (shown in [Appendix G) applying RGSC to MuZero one of the Atari games, Pac-Man, show that under the same training budget, RGSC-MuZero reaches 5166 points, compared to 3704 for MuZero. This demonstrates the poten- tial of RGSC to improve learning efficiency beyond board games. Future work extend RGSC to more domains, such as stochastic environments (Antonoglou et al. 2021) and continuous control tasks (Hubert al. 2021).
The regret network also provides interpretability by revealing specific weaknesses in the agent's learning. Furthermore, the ability of RGSC to improve even on well-trained model indicates its scalability to more complex environments such as 19x19 Go or large-scale sequential decision- making problems. We believe RGSC is promising direction for advancing RL field.
ETHICS STATEMENT
We do not foresee any ethical issues in this work.
REPRODUCIBILITY STATEMENT
To reproduce this work, provided the details of the algorithm in Appendix B and the hyper- parameters in AAppnddix A The source code, trained models used in the experiment, along with a README file and trained models are available https://rlg..iss https/iiiiiacu.../aapesss/eeeeee
THE USE OF LARGE LANGUAGE MODELS (LLMs)
Large language models (LLMs) were used only for grammar correction and proofreading in the preparation of this paper.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEGEMENT
This research is partially supported by the National Science and Technology Council (NSTC) of the Republic of China (Taiwan) under Grant Number NSTC 113-2221---0000000-M11, NSTC 113- 2634-E-A49-004, and NSTC 114-2221-E-44--005.
REFERENCES
Ioannis Antonoglou, Julian Schrittwieser, Sherjil Ozair, Thomas K. Hubert, and David Silver. Plan- ning in Stochastic Environments with a Learned Model. International Conference on Learning Representations, October 2021.
Jónsson Björnsson, Y. Expediting Self-Play Learning in AlphaZero-Style Game-Playing Agents. In Proceedings of the 26th European Conference on Artificial Intelligence, pp. 263-270. IOS Press, 2023.
Cameron B. Browne, Edward Powley, Daniel Whitehouse, Simon M. Lucas, Peter I. Cowling, Philipp Rohlfshagen, Stephen Tavener, Diego Perez, Spyridon Samothrakis, and Simon Colton. A Survey of Monte Carlo Tree Search Methods. IEEE Transactions on Computational Intelligence and Al in Games, 4(1):1-43 March 2012.
Zhendong Chu, Renqin Cai, and Hongning Wang. Meta-Reeinorrcement Learning via Exploratory Task Clustering. Proceedings of the AAAI Conference on Artificial Intelligence, 38(10): 1633- 11641, March 2024.
Rémi Coulom.
Efficient Selectivity and Backup Operators Monte-Carle Tree Search. In Com- puters and Games, Lecture Notes in Computer Science, pp- 72 83, Berlin, Heidelberg, 2007.
Springer.
Michael Dennis, Natasha Jaques, Eugene Vinitsky, Alexandre Bayen, Stuart Russell, Andrew Critch, and Sergey Levine. Emergent Complexity and Zero-shot Transfer via Unsupervised Environment Design. In Advances in Neural Information Processing Systems, volume 33, pp. 13049-13061. Curran Associates, Inc., 2020.
Adrien Ecoffet, Joost Huizinga, Joel Lehman, Kenneth O. Stanley, and Jeff Clune. First return, then explore. Nature, 590(7844):580-556 February 2021.
Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Picter Abbeel. Reverse Cur- riculum Generation for Reinforcement Learning. In Proceedings of the 1st Annual Conference Robot Learming, pp. 482-495. PMLR, October 2017.
Abhishek Gupta, Benjamin Eysenbach, Chelsea Finn, and Sergey Levine. Unsupervised Meta- Learning for Reinforcement Learning, April 2020.
Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recogni- tion, pp. 770-778, 2016.
Shih-Chieh Huang, Broderick Arneson, Ryan B. Hayward, Martin Müller, and Jakub Pawlewicz. MoHex 2.0: A Pattern-Bar Basted MCTS Hex Player. In Computers and Games, volume 8427, pp. 60-71. Springer International Publishing, Cham, 2014.
Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, and David Silver. Learning and Plamning in Complex Action Spaces. In Proceedings of the 38th International Conference Machine Learning, pp. 4476 4486. PMLR, July 2021.
Donald Joseph Hejna lí and Dorsa Sadigh. Few-Shot Preference Learning for Human-in certan--i-tth---oon -nop RI. In Proceedings of The 6th Conference on Robot Learning, pp. 2014-2025. PMLR, March 2023.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized Level Replay. In Proceedings of the 38th International Conference Machine Learning, pp. 4940 4950. PMLR, July 2021.

---

Published as a conference paper at ICLR 2026
Xiaozhen Niu, Akihiro Kishimoto, and Martin Müller. Recognizing Seki in Computer Go. In Ad- vances in Computer Games, Lecture Notes in Computer Science, pp. 103, Berlin, Heidelberg, 2006. Springer.
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving Curricula with Regret-Based Environment Design. In Proceedings of the 40th International Conference on Machine Learning, pp. 17473-17498. PMLR, June 2022.
Eric Piette, Dennis J. N. J. Soemers, Matthew Stephenson, Chiara F. Sironi, Mark H. M. Winands, and Cameron Browne. Ludii The Ludemic General Game System. Proceedings of the 24th European Conference Artificial Intelligence, pp. 41 1-418. IOS Press, February 2020.
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go. chess and shogi by planning with a learned model. Nature, 588(7839):604-606 60960004699,,, December 2020.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, Yutian Chen, Timothy Lillicrap, Fan Hui, Laurent Sifre, George van den Driessche, Thore Graepel, and Demis Hassabis. Mastering the game Go without human knowledge. Nature, 550(7676)3354-35)) 359, October 2017.
David Silver, Thomas Hubert, Julian Schrittwieser, loannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Si- monyan, and Demis Hassabis. A general reinforcement learning algorithm that masters chess, December 2018.
shogi, and Go through self-play, Science, 362(6411):
Richard S. Sutton. Dyna, an integrated architecture for learning. planning, and reacting. SIGART Bull., 160):160,,63 1991.
Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. Adaptive Com- putation and Machine Learning Series. MIT Press, Cambridge, MA, USA, edition, November 2018.
Arash Tavakoli, Vitaly Levdik, Riashat Islam, Christopher M. Smith, and Petar Kormushev. Explor- ing Restart Distributions, August 2020.
Alexandre Trudeau and Michael Bowling. Targeted Search Control in AlphaZero for Effective Pol- icy Improvement. In Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, AAMAS '23, pp. 842-850, Richland. SC, May 2023, International Foun- dation for Autonomous Agents and Multiagent Systems.
Pedro A. Tsividis, Joao Loula, Jake Burga, Nathan Foss, Andres Campero, Thomas Pouncy, Samuel J. Gershman, and Joshua B. Tenenbaum. Human-Level Reinforcement Learning through Theory-Base Based Modeling, Exploration, and Planning, July 2021.
Ikechukwu Uchendu, Ted Xiao, Yao Lu, Banghua Zhu, Mengyuan Yan, Joséphine Simon, Matthew Bennice, Chuyuan Fu, Cong Ma, Jiantao Jiao, Sergey Levine, and Karol Hausman. Jump-Start Reinforcement Learning. In Proceedings of the 40th International Conference on Machine Learn- ing, pp. 34556-34583. PMLR, July 2023.
Jane X. Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z. Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn, January 2017.
David J. Wu. Accelerating Self-Play Learning in Go. In Proceedings of the AAAI Workshop on Reinforcement Learning in Games, November 2020.
Ti-Rong Wu, Hung Guei, Pei-Chiun Peng, Po-Wei Wei Huang, Ting Han Wei, Chung-Chin Shih, and Yun-Jui Tsai. MiniZero: Comparative Analysis of AlphaZero and MuZero on Go, Othello, and Atari Games. IEEE Trunsactions on Games, 17(1):125-131 March 2025.
12

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS
A.1 TRAINING RGSC IN BOARD GAMES
In this section, we describe the details for training models used in the experiments. The training in section 4|is conducted on machine with two Intel Xeon Silver 4516Y+ CPUs and four NVIDIA RTX A6000 GPUs. For the implementation Go-Exploit, are implemented based on an open-sourced AlphaZero framework (Wu et al.) 2025). During training, 4 self-play workers and I optimization worker are used. For RGSC, each worker maintains its prioritized regret buffer (PRB).
RGSC, the two baseline networks, AlphaZero and
In subsection 4.2 we use hyperparameters shown in Table to train all three methods. For the training of RGSC, the additional hyperparameters set as follows: the sampling probability À for sampling from the buffer is set to 0.5, the buffer sampling temperature 0.1, the buffer size is 100, and the EMA coefficient is 0.5.
Table 2: Hyperparameters for training from scratch (sussection 4.2 subsection 4.3
and from well-trained model
from scratch
from well-trained
Parameter
Go
Hex
Othello
Go
Board size
11
Optimizer
SGD
SGD
SGD
SGD
Optimizer: learning rate
0.02
0.02
0.02
0.001
Optimizer: momentum
0.9
0.9
0.9
0.9
Optimizer: weight decay
0.0001
0.0001
0.0001
0.0001
MCTS simulation
200
200
200
400
Softmax temperature
1
1
Iteration
300
300
300
40
Self-Play states per iteration
160,000
120,000
120,000
160,000
Optimizations per iteration
200
200
200
1000
Batch size
1024
1024
1024
256
# Residual blocks
3
15
# Residual blocks filters
256
256
256
128
Replay buffer size
20
20
20
Dirichlet noise ratio
0.25
0.25
0.25
0.25
In subsection 4.3] we train all three methods from already well-trained models with the hyperpa- rameters shown in Table 2 In this training setup, the buffer size is 500. To warm up the PRB in RGSC, we generate 1,000 self-play games with a sampling probability X = 0.5. These warm-up games are only used in the PRB, and excluded from the training data.
A.2 COMPUTATIONAL COST ANALYSIS
Although RGSC adds two additional heads, the regret value head and the ranking head, they share the same backbone the policy and value network, so the additional computation minimal, especially in larger models. To quantify this, we measured both the neural network inference time and the per-iteration wall-clock time on Go for the 3-block model (used in subsection 4.2) and the 15-block model (used subsecctti 4.3). The results are listed in the Table 3 For the 3-block model, RGSC is 1.35x slower in inference and 1.25x slower per iteration compared to AlphaZero or Go- Exploit. However, for the 15-block model, RGSC is approximately 1.03x slower in both inference realistic and per-iteration compared to AlphaZero and Go-Exploit, which are nearly identical.
setting for past research, AlphaZero used 20 blocks, and KataGo (Wu] 2020) used 20 to 40 blocks, so the overhead becomes negligible. Therefore, RGSC adds almost no extra cost while improving training efficiency in practice.
13

---

Published as a conference paper at ICLR 2026
Table 3: Forwarding time across different blocks of the network.
blocks
3
6
9
12
15
18
20
RGSC
139.5%
109.19 1%
105,4%
103.5%
102.7%
101.8%
101.9%
AlphaZero
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
100.0%
A.3 EVALUATION
For evaluating each method, repeat each experiment twice per game and average the results to minimize stochastic variance. The number MCTS simulations is set to 400, with the softmax temperature for the final action selection.
A.3.1
EVALUATION AGAINST THE ALPHAZERO
For the experiments Figure 4 of (ubsectioon 4.2 and Figure of subsection 4.3 all methods evaluated against the AlphaZero baselines every 15 iterations, with 200 games played per evaluation.
A.3.2 EVALCATION AGAINST OPEN SOURCE PROGRAMS
In subsection 4.2 and subsection 4.3 we evaluated all the methods against open-sourced programs across all three games. The setup of evaluations for each program is outlined below:
KataGo. For the evaluation of 9x9 Go in Table I of
|sfusssctiionnn 4.2 we selected KataGio (Wu] 2020)
models from ID 1 to listed Table 4] as baselines. For each baseline, conduct 200 evaluation games with 100 games as Black and 100 games as White, with the simulation count for KataGo fixed at 400.
For the evaluation against KataGo in subsecttio 4.3 we pick KataGo with ID 5 in Table 4 In
this experiment, actions are selected without applying softmax, and 1,200 evaluation games are conducted for each method.
Table The versions of the selected KataGo models for 9x9 Go.
ID
Version
# blocks
Elo ratings
9833.3 + 16.1
2
9900.6 + 16.2
3 kataa=#66B-337759922222888
9958.6 + 16.9
4 kataa-6boeer2---444188888-------
10
10138.6 6 + 18.3
5 kataa-1bboc112--88777777---
15
11180.1 + 16.1
Alpha-Beta Algorithm in Ludii. For the evaluation of 10x10 Othello in Table Jubbseciiooo 4.2 we use the Alpha-Beta algorithm from Ludii (Piette et al. (0200) with search levels set to 2, 3, and 4 as our baselines. For each method in Table a total of 300 games played, with 150 games as
Black and 150 games as White.
MoHex. For the evaluation of 1xx11 Hex Table I of /subsection 4.2, all methods fight against MoHex (Huang et al.) 2014). For the MoHex setup, the maximum thinking time is set to I second, without using a cache book. Additionally, we use two MoHex baselines with the following search settings: a search width of 15 with maximum search depth of 5, and search width of 25 with maximum search depth of 8.
B RGSC ALGORITHM
In this section, we describe the details of the RGSC algorithm in Algorithm contains a value network fregret, regret ranking network frank within the prioritized regret buffer (PRB). Lines 4 speciif the procedure of buffer sampling, while I3 outline the self-play process, during which all nodes in the search tree evaluated by the regret network and describe how opening
14

---

Published as a conference paper at ICLR 2026
describe how opening is updated and how it is inserted into the buffer. In this procedure, search tree nodes are inserted into the PRB, enabling us to exploit the search tree while exploring previously unseen states with potentially high regret.
is selected. Line 31 35
Algorithm RGSC Algorithm
Require: buffer 8, buffer size N, buffer rate X, buffer sampling rule (33
Require: regret ranking network frank, regret value network fregret.
1: Initialize buffer 3
2: while self-play do
3: Reset the environment to the initial state so
4: Sample random number p E (0,
5: if p < À then
6: Sample opening o from 8 with Vi (3)
7: Set o as the starting state
8: else
9: Set so as the starting state
10:
end if
11:
Initialize global candidate state s' < 0, ranking score 7 + -0, regret value g' + --0
12:
Self-play from the starting state
while the environment state 8, is not terminal do Run MCTS current state 8, and select action
14:
15:
for each search node s the MCTS tree do
16:
Predict ranking score Ys Predict regret value 9.
17:
18:
if 7 > then
19:
t 7 + 7si g' + Is
20:
end if
21: end for
Execute a, observe next state 8t-1- set t + t t 1, 81 + 81+1
23:
end while
for each 8, in the pelf-play trajectory Snt,,,, do
25:
Use final return to compute regret R(::)
26:
Get ranking score Ys.
27:
if Toc 7 then
28:
s' St: 7' t 7s.
g'RR(s.
29:
end if
30:
end for
31:
if p then
32:
Update the regret of opening o in buffer 3 using the EMA rule in Equation 13
else
34:
Store the candidate s' with its regret g' into buffer /
end if
36:
end while
C DETAIL FOR TOY MODEL EXPERIMENT
In the experiments in inssubecttoon 4.1 we implemented a simple Q-learning example on sparse-reward binary trees with five and six levels. In Q-learning, the learning rate is set to 0.1, is set to 0. in epsilon greedy, and the discount factor y also set to 0.1. We compare the training speed of three opening sampling strategies for selecting the starting state in self-play. The first one always starts from the root. The second one uniformly samples a non-terminal state from a fixed-size first-in- first-out buffer that stores past trajectories with buffer rate of 0.5, similar to the GEVC method in Go-Exploit. The third strategy, the regret-based method, samples states in proportion to their regret values. For the random method and regret method, buffer rate of 0.5 is applied. Each method is trained for 6,000 iterations. For every 100 iterations, 6000 games are played for evaluation, and the average reward is calculated. To reduce the variance, use 25 different seeds in the experiments.
15

---

Published as a conference paper at ICLR 2026
In Figure 3b compare the difference between the root's estimated Q Q-value and its theoretical optimal Q-value across the three methods.
D PERFORMANCE UNDER DIFFERENT HYPERPARAMETER SETTINGS IN RGSC
In this section, we explore the hyperparameters in RGSC, including the sampling probability (A), the buffer sampling temperature (r), the buffer size (x), and the EMA coefficient (a) across 9x9 Go, 10x10 Othello, and 11x11 Hex. In the ablation study, we aim to choose the setting with less computational cost and better performance. Additionally, we select the setting that demonstrates stable and consistent results across different games.
D.I SAMPLING PROBABILITY
We evaluate different sampling probabilities (A), which represent the probability (as introduced in Section 3.3) of starting a self-play trajectory from state sampled from the PRB. We evaluate E (0.2, 0.5, 0.9}, as shown in Figure 9 Overall, RGSC with
0.5 performs consistently well across the three games, so we use x 0.5 for our final setting. In 1 1xx1 Hex (Figure 9c). a high sampling probability (A = 0.9) yields an early improvement around 10K training steps, but subsequently causes significant performance instability toward the end of the training process. Moreover, in 9x9 Go, also shows that À = 0.9 exhibits marginally reduced stability (Figure 9a). These may be due to overly frequent sampling of recent self- self-play data from the PRB.
1409
1000
1250
1400
1200
1000
1200
Eio Ratinp
Ele Rating
Pating
820
a00
630-
500
1-0.2
409
A-02
A-02
200
1.0.5
200
A0055
2505
1e0.9
1-2
1-09
60k
20k
5tK
Training Steps
Training Steps
Training Steps
(a) 9x9 Go
(b) 10x10 Othello
(c) 11x11 Hex
Figure 9: Playing performance of different sampling probabilities (A) across three games. The orange curve is the final RGSC setting.
D.2 BUFFER SAMPLING TEMPERATURE
Furthermore, we investigate the effect of the buffer sampling temperature (r) described in Subsec- tion 3.3] where a higher value leads to more uniform softmax distribution for the PRB. We test E (0.1, 1, 0.5, 1}. as shown in Figure 10 and the results show that T 0.1 achieves the best performance.
D.3 BUFFER SIZE
We evaluate different buffer sizes, x E (100, 500, 1000) in RGSC, as shown in Figure 11 The results show significant difference in performance, minimize computational cost.
100 in our final setting to
D.4 EMA COEFFICIENT
For the EMA coefficient used to update the regret values in Equation 13 of the main text, test E {0.1, 0.5, 1), as shown in Figure 12 finding that a 0.5 generally yields the best performance in RGSC. Note that only uses the regret calculated from the newly finished self-play game,
16

---

Published as a conference paper at ICLR 2026
1400
1400
1000
1220
1200
1400
Elo Rating
1030
Elo Rating
1000
Rating
1200
1.000
630-
420
1.0.5
202
200
1505
[-]
[=]
10K
20C
S0K
40K
50k 6k
6K
10k
23K
30K
40K
50K
60k
106
20K
30K
43K
50k
50c
Training Steps
Training Steps
Training Steps
(a) 9x9 Go
(h) 10x10 Othello
(c) 11x11 Hex
Figure 10: Playing performance of different buffer sampling temperatures (T) across three games. The orange curve the final RGSC setting.
1450
1000
1200
1400
1000
1200
Elo Rating
a20
Rating
1000
803
di Ratiin
1000
400-
402
400
200
200
--000
1000
10k
20C
S5K
40k
50k
10K
0K
10c
20K
30k
40k
50k
60k
06
10k
20K
30c
43k
50K
50c
Training Stops
Training Stops
Training Steps
(a) 9x9 Go
(b) 10x10 Othello
(c) 11x11 Hex
Figure 11: Playing performance of different buffer sizes (I)) across three games. The orange curve the final RGSC setting.
explaining its slightly unstable performance in 10x10 Othello (Figure 12b and 11x11 Hex (Figure 12cb.
In conclusion, these experiments show that RGSC is not highly sensitive hyperparameter choices and that our recommended configuration works well across different games, demonstrating the over- all robustness of the method.
E ANALYSIS OF
STATES
To further examine the high-regret states stored in the regret buffer, analyze several examples that RGSC frequently selects in 9x9 Go, 10x10 Othello, and 11x11 Hex. In Figure 13, we show examples of high-regret states sampled from the regret buffer of RGSC in these three games. To evaluate whether the agent has learned the technique for solving these high-regret states, we evaluate the RGSC agents against an AlphaZero baseline, using this state the starting position, as shown in Figure 14 Finally, we track the evolution of regret values and corresponding game outcomes for these examples throughout training, showing their relationship in Figure 15)
9x9 Go. In Figure 13a White can only win by keeping the stones at I8 (A) alive. Once White plays H9 (B), a seki (Niu et al., 2006) arises at the top-right corner. In a seki, neither player can capture the other's stones-the first player play inside the seki area will have their stones captured. Thus, forming the seki the only path to victory for White. As shown Figure 14a] the win rate for this example increases from 47% to over 90% within few training iterations, indicating that RGSC successfully learns the strategy required to win from this high-regret state.
To further verify whether the agent has truly mastered this high-regret situation, we evaluate a 9x9 Go state (Figure 13b) that shares the same structural pattern as the one in Figure 13a where White can only win by playing at H9 (C). In Figure 14a the agent's win rate in this state increases from 47% to 85%. This shows that RGSC enables the agent to identify weaknesses in its current policy and correct them automatically, and generalize its learned strategies structurally similar situations.
17

---

Published as a conference paper at ICLR 2026
1400
1400
1600
1220
1200
1400
Rating
1030-
Elo Rating
1000
Ratine
1200
630-
420
200.5
200
0555
g-1
0-1
10K
20C
30K
40K
50k
10k
25K
30K
40K
50K
60K
106
20K
30K
43K
50k
50c
Training Steps
Training Steps
Training Steps
(a) 9x9 Go
(h) 10x10 Othello
(c) 11x11 Hex
Figure 12: Playing performance of different EMA coefficients (a) across three games. The orange curve the final RGSC setting.
3
(a) 9x9 Go (White's Turn)
(b) A state similar to (a) (White's Turn)
10
(c) 10x10 Othello
(d) 11x11 Hcx
(Black's Turn)
(Blue's Turn)
Figure 13: High-regret state examples. Blue triangles indicate the last moves.
10x10 Othello. A high-regret example of 10x10 Othello is shown in Figure 13c where Black secures guaranteed by playing J1 (D) followed by J10 (E). 13K training steps, the agent's 's win rate for Black at this example is 82%, as shown in Figure 14b After the first update, no significant improvement was observed. However, following the second update at 13.8K training steps, the win rate sharply increases to 100%. This indicates that RGSC has successfully Icarned the correct strategy for this high-regret state and no longer makes mistakes when encountering it.
11x11 Hex. Finally, high-regret example of 1xx1 Hex is shown in Figure 13d Blue can win by playing F2 first (F), followed by D2 (G). After the first update at 18.4K training steps, the win rate for this example surges to 88%. shown in Figure [4c] After additional round of training at 19.0K steps, the win rate further increases to 91 %, showing a 14% improvement within single iteration. These results demonstrate that RGSC not only enables the agent correct
In this example,
18

---

Published as a conference paper at ICLR 2026
1.0
8.-
Wincate
8.8-
Winrate
0.85
2.-
0.00
6.75
High-legrel State Similar
3.
High-legre:
High
3.4
24.55
24.5K
25.3K Trinning Steps
25.2K
18.0K 13.2 13.40 13 14,0K 34.2K JA Training Steps
184c
18.0K
18.K
15.K
Training Steps
(a) 9x9 Go
(b) 10x10 Othello
(c) 11x11 Hex
Figure 14: Win rates of high-regret states during RGSC training.
its previous mistakes but also enhances the interpretability of the AlphaZero framework by revealing how the model progressively learns to resolve high-regret states.
12-
Mile
Whete Win
3.33
62
2444
30.0
ecc
163
186
lLA4
Training
(a) 9x9
(b) 10x10 Othello
(c) 11x11 Hex
Figure 15: Relationship between game outcomes and regrets during training.
Finally, we look into the relationship between the game outcomes and the corresponding regrets of these examples throughout the training, as shown Figure 15) For state where White always wins under the optimal play, the regret associated with White's victory decreases the agent's predictions on that state become more accurate. We first examine the examples in 9x9 Go and 11x11 Hex, where White (represented as Red in Hex) can win under the optimal policy. As shown in Figure [5a and Figure [5c] the regret for White's s win diminishes with more training iterations on these openings, while the regret for Black's win increases correspondingly. On the other hand, examine the example in 10x10 10 Othello, where Black has a guaranteed win, as shown Figure [3b] The average regret values associated with Black's wins decrease training progresses. By 14.2K training steps, no outcomes of White wins are observed, showing that the agent has fully learned the correct strategy win in this example. In conclusion, the convergence regret indicates that the agent is progressing toward the optimal policy during training.
F ANALYSIS OF PRIORITIZED REGRET BUFFER
In this section, we investigate the openings' attributes in PRB. In subsection F. we investigate the distribution opening lengths during training and how the agent adapts to different phases of the game. Next, in subsection F.2, track how the regret values of the openings evolve throughout training, showing that the agent becomes increasingly familiar with high-regret states and progres- sively refines its policy.
OPENING-LENGTH DISTRIBUTIONS IN TRAINING PROCESS
In board games such as Go, Hex, and Othello, the game is typically divided into three phases: early game, midgame, and endgame. Among these, the midgame typically involves the most complex tactics and combinatorial challenges, offering the greatest potential for learning. To which phase
19

---

Published as a conference paper at ICLR 2026
of the game is preferred for learning during training, examine the length of openings used in PRB throughout the training process. In Figure 16 Figure 17 and Figure 18 we compare RGSC with variant that excludes the regret ranking network and selects openings solely from trajectories based on computed regret.
Opering Length: O--4 Opening Length:5--3
Openiing Length: 10-19
Opening Length: 40- 59
Opening Length:
Opening Length
100%
100%
80%
proportion
Proportion
0094
40%
20%
20
300
125 140
220
Iteration
(a) RGSC w/o regret ranking network
(b) RGSC
Figure 16: Change in the proportion of openings with different lengths across training 9x9 Go.
Opening Opening Length:
Opening Longth:
40-59
Opening Longth: 20-39
Opening Langth
100%
120%
proportion
60%
Proportion
10%
40%
0%
20
60
60
60
150
130
160
140
200
220
240
260
200
300
0%
20
40
80
100
120
140
140
1éa
200
220
240
260
200
300
Iberation
Beration
(a) RGSC w/o regret ranking network
(b) RGSC
Figure 17: Change in the proportion of openings with different lengths across training in 10x10 Othello.
Opening Length: 0-4
Opening Longth: 10-19
Opening Length: 40-59
Opening Lengths
Opening Length: 20-39
Opening Length
100%
120%
Proportion
60%
Proportion
40%
40%
20%
0%
150 120 160 20
200
220
240
260
300
40
80
120
140
180
220
260
280
350
Iteraaton
(a) RGSC w/o regret ranking network
(b) RGSC
Figure 18: Change in the proportion openings with different lengths across training 1Ix11 Hex.
Figure 16 shows the dynamics of the proportion of openings with different lengths throughout the training process RGSC with and without a regret ranking network in 9x9 Go. RGSC without
regret ranking network shows significant preference for opening lengths, shown in Figure
Toal In contrast, RGSC exhibits a clear phase-aware curriculum. In the early stages training, favors openings with opening lengths ranging from 20 to 39, corresponding to midgame positions,
20

---

Published as a conference paper at ICLR 2026
which are the most complex and informative. As training progresses and the agent's overall strength improves, so that games reach midgame phases earlier, the distribution shifts toward shorter opening lengths, indicating that RGSC continues to target the most inpborationwortlyy states the MCTS value estimation improves.
Figure 17 and Figure 18 show the results of the analysis in Hex and Othello. We observe the same qualitative pattern: relative to the RGSC without regret ranking network, RGSC progressively biases toward shorter opening lengths training advances. The consistency across various games indicates that RGSC selects openings based on the model's playing strength, allowing it to identify the most valuable learning states different training stages.
The dynamics of opening lengths show that RGSC induces an implicit, data-driven curriculum on game phases: initially focusing on complex midgame positions, then shifting shorter openings as the model's playing strength improves. In contrast, RGSC without regret ranking network remains less sensitive the phases and difficulty of the game.
F.2 DECREASING REGRET VALUES ACROSS TRAINING
14-
10
Fatt
2nd
215
600
Fre
218
2nd
2:1
45
2/h
(a) 9x9 Go
(b) 10x10 Othello
(c) 11x11 Hex
Figure 19: Changes in the values of openings from their first appearance to the final update during training.
We analyze how the regret values of openings stored in the regret buffer evolve during training. Specifically, we track the regret values of openings from their first appearance to their final removal, capturing their updates across five training iterations. As shown in Figure 19, the regret values consistently decrease with each update, indicating that the model becomes increasingly familiar with these high-regret openings. By the final update, the overall regret distribution shifts toward lower values, resulting in the lowest regret levels observed across all iterations. This demonstrates that the agent progressively focuses challenging states and refines its policy to reduce the associated regret over time.
G RGSC ON ATARI GAMES
Returns
4K
2K
MuZero
RGSC
0K
20K
40K
60K
Training Steps
Figure 20: Playing performance of MuZero, and RGSC on Ms. Pac-Man. The shaded area a 95% confidence interval for the mean.
21

---

Published as a conference paper at ICLR 2026
RGSC can also be generalized to MuZero (Schrittwisser et al. 2020) other AlphaZero Alphazero-variants To demonstrate this, we further integrate RGSC into MuZero in a large-scale domain, the Atari benchmark. Specifically,
select one of the Atari games, Ms. Pac-Man, in evaluation. Unlike board games, where the outcome is only available at the terminal state, we use the internal rewards compute n-step return for in |Equation for all intermediate states in Atari games.
Since Go-Exploit is implemented only within AlphaZero for board games, our experiment focuses on comparing RGSC with MuZero. Throughout the entire training process, RGSC significantly outperforms MuZero, as shown in Figure 20) At the final iteration, RGSC achieves an average score of 5166 points, while MuZero only achieves 3704 points, demonstrating the generality of RGSC when applied to other AlphaZero-like style algorithms and highlighting its ability to handle complex tasks.
22


Reviews:
[{"review_id": "ixcorjV8C9", "reviewer": "Reviewer_CfZM", "summary": "The paper introduces Regret-Guided Search Control (RGSC), an extension to AlphaZero that enhances learning efficiency in reinforcement learning for board games by prioritizing restarts from high-regret states—positions where the agent's evaluation significantly diverges from the actual game outcome. RGSC incorporates a regret network (comprising a ranking network and a value network) to identify these states from self-play trajectories and MCTS nodes, storing them in a prioritized regret buffer for reuse as starting points.\n\nContributions include a novel regret-based ranking objective to handle imbalanced and non-stationary regret distributions, a toy example demonstrating regret-guided benefits, and empirical results showing RGSC outperforms AlphaZero and Go-Exploit by 77 and 89 Elo on average across 9x9 Go, 10x10 Othello, and 11x11 Hex. Additionally, RGSC improves performance on nearly converged models (e.g., boosting win rate against KataGo from 69.3% to 78.2%), highlighting its robustness.", "questions": "1. Given the fluctuations and closeness in Figure 4's Elo curves (e.g., overlaps in confidence intervals), how stable is RGSC's advantage? Could you report results from extended training (e.g., to 70K-100K steps) or more random seeds to assess if the lead holds or diminishes over time? \n2. Same question for Figure 6.\n3. Could you report the relative computational cost of RGSC (e.g., average time per self-play game or optimization step) compared to AlphaZero and Go-Exploit? How much overhead does the regret network introduce, and does it offset the sample efficiency gains?\n4. The toy example uses a simple regret ($|\\hat Q(s) - Q(s,a)|$); how does this compare to the definition in Equation 2?\n5. I find the RGSC method to be very strong. Could the authors discuss the potential for applying RGSC in more complex domains, specifically those involving stochastic dynamics or partial observability? What challenges would the regret network face in these settings?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper demonstrates strong originality by creatively combining search control concepts (e.g., from Sutton & Barto, 2018, and Go-Exploit) with a novel regret-guided prioritization mechanism tailored to AlphaZero. This includes a ranking-based objective for the regret network, which addresses challenges like imbalance and non-stationarity in regret prediction—issues not fully tackled in prior work like Tavakoli et al. (2020) or Trudeau & Bowling (2023). While building on existing ideas, the application to identify high-impact states in MCTS trees and self-play trajectories removes limitations of uniform sampling, offering a fresh formulation inspired by human learning patterns.\n\nIn terms of quality, the work is rigorous, with comprehensive experiments on three diverse board games using fixed training states for fair comparison, substantiated by Elo ratings, error bars, and additional analysis. The methodology includes derivations and pseudocode in appendices, enhancing reproducibility.\n\nClarity is excellent; the paper is well-structured, with intuitive figures.\n\nSignificance is high, as RGSC could extend to broader RL domains (e.g., robotics or planning) where sample efficiency is critical.", "weaknesses": "1. Adding the regret network increases model complexity (e.g., extra heads and training objectives), but the paper doesn't quantify the overhead in terms of GPU hours or inference time during self-play. This could be addressed by reporting relative costs compared to baselines, ensuring the efficiency gains aren't offset by higher per-iteration expenses.\n\n2. While RGSC shows final Elo improvements, the Elo curves in Figure 4 reveal that advantages are not consistently stable across training, with baselines often close and fluctuations (e.g., dips around 50K-60K steps) raising concerns about robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MnocrzqtwP", "reviewer": "Reviewer_taWz", "summary": "The paper proposes the RGSC, which introduces a regret network to identify high-regret states where the agent's evaluation significantly deviates from the actual game outcome. And experiments are conducted on 9×9 Go, 10×10 Othello, and 11×11 Hex.", "questions": "1. Can RGSC be generalized to other benchmarks such as Atari or robotic control tasks where terminal outcomes are ambiguous or partial observability is present?  \n\n2. The method uses many hyperparameters. Have these been validated via ablation studies?", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper defines “regret” as the average cumulative deviation from the current state to terminal states. This formulation captures the long-term impact of mistakes while avoiding the locality limitations of single-step error measures. \n\n2. The method is evaluated on several board-game domains, with a reasonably comprehensive experimental suite within that problem class.", "weaknesses": "well RGSC generalizes beyond these discrete, perfect-information domains. \n\n2. Dependence on terminal observability: RGSC’s regret computation requires access to complete state-to-terminal outcomes. In many real-world or continuous-control tasks (e.g., robot, Atari), the notion of a single terminal outcome can be ambiguous or trajectories cannot be fully recovered, making the regret signal difficult or impossible to compute reliably. \n\n3. Baselines: the baselines appear outdated (2023 as the most recent); stronger, more recent baselines should be included to better position RGSC relative to the current state of the art. \n\n4. Hyperparameter ablation: the method involves many hyperparameters, but the paper lacks ablation studies to show sensitivity and justify the chosen settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HheEvYPzPO", "reviewer": "Reviewer_4XJP", "summary": "This paper proposes Regret-Guided Search Control (RGSC) to improve the efficiency of AlphaZero-style reinforcement learning. RGSC uses a regret network to identify high-regret states, positions where the agent’s evaluation most diverges from the outcome, and stores them in a prioritized buffer for replay as new starting positions. Experiments on 9×9 Go, 10×10 Othello, and 11×11 Hex show RGSC outperforms AlphaZero and Go-Exploit by 77 and 89 Elo points on average, and it improves win rates even from strong pre-trained models. Overall, RGSC provides an effective mechanism for search control, enhancing both the efficiency and robustness of AlphaZero training.", "questions": "- Would RGSC remain effective on large-scale domains like 19×19 Go or Atari, where regret estimation might be noisier?\n- How significant is the computational overhead introduced by the regret network compared to standard AlphaZero training?\n- Could the regret signal be estimated directly from existing AlphaZero components (e.g., policy/value mismatch) without training a separate network?\n- How sensitive is RGSC’s performance to the design of the ranking-based loss or the regret buffer prioritization mechanism?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Novel and well-motivated idea: The use of regret-guided search control is an intuitive and principled extension of AlphaZero that better mimics human-style targeted learning.\n- Clear technical contribution: Introducing a regret network and a ranking-based objective to identify and prioritize high-regret states is a concrete and original addition to the AlphaZero framework.\n- Strong empirical results: RGSC shows consistent performance improvements across multiple domains (Go, Othello, Hex) and against strong baselines (AlphaZero, Go-Exploit).\n- Robustness to training stage: The method continues to yield improvements even when applied to well-trained models, suggesting it generalizes beyond early training efficiency.", "weaknesses": "- Limited theoretical justification: The paper mainly relies on empirical validation; it lacks a clear theoretical analysis of why regret-based prioritization should improve sample efficiency.\n- Scalability concerns: Results are shown only on small board sizes (9×9 Go, 10×10 Othello, 11×11 Hex). It’s unclear if RGSC scales to larger or more complex domains (e.g., 19×19 Go).\n- Computational overhead: Maintaining a regret network and prioritized buffer might add non-trivial computational cost; efficiency trade-offs are not clearly discussed.\n- Ablation detail: The paper would benefit from more ablation or sensitivity analysis (e.g., effect of buffer size, ranking loss design, or sampling strategy).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YtvtH6GWwn", "reviewer": "Reviewer_WKT8", "summary": "The paper proposes a new technique for improving the efficiency of training of MCTS agents by choosing which states to start playing from, rather than always starting from the very beginning (AlphaZero) or from uniform sampling of visited states (Go-Exploit).\n\nThe authors call this method Regret-Guided Search Control (RGSC). It prioritizes sampling states that have high regret, where regret is the discrepancy between the agent's evaluation and the game's actual outcome. To predict such regrets, they train both a regret value network and a regret ranking network. The predictions are then used to select states from a prioritized regret buffer (PRB).\n\nThe authors evaluate on a number of games, including Go, Othello, and Hex. They do an ablation study to see whether the regret ranking network actually helps. They also measure whether high-regret states in the PRB gradually decrease during training.", "questions": "Page 1:\n\n\"represented as cells\" What does this mean?\n\nPage 4:\n\nA ranking-based objective might partially address non-stationarity, but can't the ranking (relative order) also change with further training?\n\nPage 5:\n\nWhy does the exponential transformation aid optimization? If the aim is non-negativity, why not use softplus or squareplus, which are more stable and less susceptible to overflow for large values?\n\nHow do you choose the sampling temperature τ, the EMA rate α, and the probability λ in a principled way in practice?\n\nAppendix 1 describes the hyperparameter values that were chosen. Were these obtained via trial-and-error? How sensitive is the performance to these choices?\n\nThe design of outputting the ranking score and getting the resulting restart distribution via a softmax seems a little ad hoc. Can you explain this?\n\nHow many trials were used for each figure (3, 4, 5, 6, 7) and table (1, 2)? Were these independent runs of the whole training process?\n\nPage 6:\n\n\"All methods use a 3-block residual network\" What activation function and weight initialization (presumably He) was used?\n\nPage 8:\n\nFigure 7 lacks uncertainty bands. Was this run for multiple trials?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This paper presents a seemingly promising idea of choosing which states to play from, rather than always starting from the very beginning (AlphaZero) or from uniform sampling of visited states (Go-Exploit). Conceptually, it makes a lot of sense that doing so could vastly increase the efficiency of the learning process.", "weaknesses": "According to the plots, the experimental results do not show a strong advantage of the proposed method in terms of Elo rating.\n\nFigure 4 does not show a strong advantage of RGSC over Go-Exploit in terms of Elo rating.\n\nFigure 5 does not show a strong advantage of RGSC over AlphaZero in terms of Elo rating.\n\nFigure 6 does not show a clear superiority of the regret ranking network over the regret value network, especially in 11 x 11 Hex. In the last case, the former *underperforms* the latter for most of the run.\n\nAlso, some choices in the design of the ranking surrogate objective have an unclear justification (more comments on this below). It would greatly help if the authors justified each step of their design, as well as how the hyperparameters would be chosen in practice.\n\nPage-by-page corrections:\n\nPage 1:\n\nlearning efficient -> learning-efficient\n\nPage 2:\n\nusing starting states -> choosing starting states\n\nlater training -> later stages of training\n\nPage 5:\n\nredundant parentheses around R(s_j)^(1/τ) in the denominator for the PRB expression\n\nPage 6:\n\nFigures 3, 4, 5, 6: \"The shaded area is the error bar with 95% confidence interval\" -> \"The shaded area is a 95% confidence interval for the mean\", if this is what you meant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)