Title: Game Solving with Online Fine-Tuning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Game Solving

Computer Games

AlphaZero

Online Fine-Tuning

Monte Carlo Tree Search

Deep Reinforcement Learning
Abstract: Game solving is a similar, yet more difficult task than mastering a game. Solving a game typically means to find the game-theoretic value (outcome given optimal play), and optionally a full strategy to follow in order to achieve that outcome. The AlphaZero algorithm has demonstrated super-human level play, and its powerful policy and value predictions have also served as heuristics in game solving. However, to solve a game and obtain a full strategy, a winning response must be found for all possible moves by the losing player. This includes very poor lines of play from the losing side, for which the AlphaZero self-play process will not encounter. AlphaZero-based heuristics can be highly inaccurate when evaluating these out-of-distribution positions, which occur throughout the entire search. To address this issue, this paper investigates applying online fine-tuning while searching and proposes two methods to learn tailor-designed heuristics for game solving. Our experiments show that using online fine-tuning can solve a series of challenging 7x7 Killall-Go problems, using only 23.54\% of computation time compared to the baseline without online fine-tuning. Results suggest that the savings scale with problem size. Our method can further be extended to any tree search algorithm for problem solving. Our code is available at https://rlg.iis.sinica.edu.tw/papers/neurips2023-online-fine-tuning-solver.

Game Solving with Online Fine-Tuning

Abstract

Game solving is a similar, yet more difficult task than mastering a game. Solving a game typically means to find the game-theoretic value (outcome given optimal play), and optionally a full strategy to follow in order to achieve that outcome. The AlphaZero algorithm has demonstrated super-human level play, and its powerful policy and value predictions have also served as heuristics in game solving. However, to solve a game and obtain a full strategy, a winning response must be found for all possible moves by the losing player. This includes very poor lines of play from the losing side, for which the AlphaZero self-play process will not encounter. AlphaZero-based heuristics can be highly inaccurate when evaluating these outof-distribution positions, which occur throughout the entire search. To address this issue, this paper investigates applying online fine-tuning while searching and proposes two methods to learn tailor-designed heuristics for game solving. Our experiments show that using online fine-tuning can solve a series of challenging 7x7 Killall-Go problems, using only 23.54% of computation time compared to the baseline without online fine-tuning. Results suggest that the savings scale with problem size. Our method can further be extended to any tree search algorithm for problem solving. Our code is available at https://rlg.iis.sinica.edu.tw/papers/neurips2023online-fine-tuning-solver.

1 Introduction

Playing and solving strategic games have served as drivers and major milestones [1] in artificial intelligence research. To master such games, the objective is often designed to optimize on the objective of maximizing the probability of winning. In the past several decades, researchers made significant progress in game playing, reaching super-human playing levels in many domains. Successful examples include Chinook (checkers) [2], Deep Blue (chess) [3], AlphaGo (Go) [4], and AlphaStar (StarCraft II) [5]. Furthermore, AlphaZero [6, 7] and MuZero [8] even boast generality by mastering a variety of games without requiring expert human knowledge. Although these learning-based agents have progressed dramatically in playing strength, there are no guarantees that their decisions are always correct [9, 10] in terms of game-theoretic value, which is defined as the outcome of the game given optimal play for both players. Game solving is this pursuit of finding game-theoretic values.

Game solving is a more difficult challenge than game playing. Many seemingly simple games have astronomically large state spaces, with no simple way of exploring this space. Here is where

advancements in game playing can aid game solving. Strong agents are commonly leveraged to evaluate positions, providing guidance and reducing the search space significantly. For example, the checkers program Chinook claimed to have reached super-human levels as early as 1996 [2], then about 10 years later, played an instrumental role in the proof that checkers is a drawn game [11]. Similarly, contemporary learning-based approaches such as AlphaZero are widely used to help reduce the search space for game solving. Game solvers that utilized AlphaZero include Hex [12], Go [13], Killall-Go [14], and the Rubik's cube [15, 16]. Such approaches are not limited to applications in games but extend to other non-game fields, like automated theorem proving [17].

However, a major issue still exists when using learning-based approaches to aid game solving. In the two-player, zero-sum setting, a simple description for a proof involves verifying that there is a winning move for the winner, for all possibilities played by the losing side; i.e. no matter how the loser plays, the winner must be able to respond correctly. However, most learning-based agents are trained along a strong line of play by both players, with some exploration to nearby states for robustness. Using AlphaZero as an example, training samples are generated via self-play by the best version of itself up to that point. Learning-based methods are powerful in that they generalize for previously unseen positions, but accuracy tends to drop the further you stray from training samples. To verify all possibilities on the losing side, the vast majority of positions we must evaluate during the search for a proof are therefore out-of-distribution. To illustrate, AlphaZero-like networks have been shown to make inconsistent or completely incorrect evaluations, simply by adding two meaningless stones to a position [10]. In another example, in the attempt to solve one of the hardest Go life-and-death (L&D) problems from the famous book Igo Hatsuyoron, 2 all AlphaZero-like programs failed. It was hypothesized that this was because these highly specific problems are rarely encountered during training [18].

This paper proposes applying online fine-tuning methods to train learning-based systems while solving games. In our proposed methods, during game solving, an online trainer is added so that the learned heuristic is improved as new data that is relevant to the solving task is generated. This is done by utilizing new information such as solved and critical positions in the current solver search tree. The trainer therefore can learn better heuristics dynamically, that are particularly fine-tuned for the upcoming evaluations. Experiments are conducted on 16 challenging 7x7 Killall-Go three-move openings, shown in Figure 3. We develop a distributed game solver with online fine-tuning, that is built upon the state-of-the-art 7x7 Killall-Go solver [13]. Experiment results show that the online fine-tuning solver can greatly reduce the search space by a factor of 4.61 on average. Namely, it searches only 21.69% of nodes, using 23.54% of the computation time, as compared to the offline solver. Most importantly, for larger problems, the online fine-tuning solver performs significantly faster than that without, which implies that our method scales with problem size.

2 Background

2.1 Game solvers

A two-player zero-sum game is considered solved if we know of a winning strategy for either player which guarantees a winning outcome, 3 regardless of how the opponent plays; i.e. the player must have at least one action that leads to a win, for all actions by the opponent. A winning strategy is often represented as an AND-OR tree called a solution tree [19], where the game positions with the winner to move are represented by OR-nodes, and those for the opponent by AND-nodes. Leaf nodes in a solution tree are all terminal positions where the outcome is a win.

A solver is a program that does a proof search and can identify a winning strategy or a solution tree, if found. Solvers often rely on heuristic tree search for games with large and complex game state spaces. Search algorithms such as alpha-beta search [20], proof number search (PNS) [21], or Monte Carlo tree search (MCTS) [22, 23] have all been shown to be successful. In addition, previous research has shown that none of the algorithms dominates the others [24, 25, 14].

2.2 Distributed game solver

In cases where search spaces are too large for a single instance solver under reasonable time and memory constraints, multiple solvers are often run in parallel, forming a distributed computing system, to scale up the solving process. Examples of games solved by distributed computing include checkers [11], heads-up limit hold'em poker [26], breakthrough [27], Hex [28], and Connect6 [25, 29].

These distributed game solving/analysis systems, also known as distributed game solvers , have been presented commonly with two components, a manager and a set of workers . A manager divides the overall problem into smaller sub-problems, keeping only the initial portion of the search tree - the beginning of the game - in memory. As this search tree is expanded, the manager may decide to offload analysis of specific positions to its workers. These offloaded sub-problems are also called jobs [25, 29]. A worker computes jobs by taking as input, a specific position and any relevant parameters (e.g. time limits), then outputs either a solved or heuristic value for that position. A worker can be a single solver, a game engine, or even a combination of both.

For example, in the checkers proof, Chinook and another depth-first proof-number search [30] solver were combined as a heuristic. From the perspective of the manager, a solved job result, such as a proven win, loss, or draw, can be thought of as a terminal node in its solution tree. Unsolved jobs also provide useful information, such as heuristic values to determine likely win or likely loss , to guide further tree expansions, with the worker acting as a relatively expensive and accurate heuristic. Similar to the checkers proof, Connect6 openings have also been solved by encapsulating the task of solving and playing a position into a single job, which was then dispatched by a manager to a set of workers [25, 29, 31]. In all examples listed above, a centralized scheme is used where one manager coordinates between dozens to hundreds of workers.

2.3 Proof Cost Network

When using neural networks as heuristics in solving, recent research points out that there is room for improvement when using the value network learned from the AlphaZero algorithm[16, 14]. In a search tree, when several actions can reach a winning outcome, AlphaZero-trained networks have no preference for choosing one that wins fastest. This can increase the amount of computation significantly.

To address this challenge, the Proof Cost Network (PCN) [14] predicts a proof cost value, rather than a win rate. The cost value represents a logarithmically-normalized estimate of the number of nodes that are required to solve the position. Specifically, PCN adopts the AlphaZero training process and generates self-play games using the cost value to guide the MCTS toward faster winning moves. These self-play games are then used to update PCN's cost values. The resulting network will focus the proof search on actions with minimal cost. Experiments show that the proof cost value is highly correlated to problem difficulty, and can significantly improve solving capability.

3 Game solver with online fine-tuning

This section describes our methods for applying online fine-tuning to game solving. We chose an MCTS-based solver due to its popularity when integrating AlphaZero networks as heuristics [16, 13, 14]. However, it is worth noting that the methods presented in this paper are searchindependent and can be readily applied to other search algorithms such as alpha-beta search or PNS.

3.1 Distributed game solver

Our distributed game solver consists of a PCN serving as its heuristic, a manager , and a set of workers . The manager maintains an MCTS rooted at the position to be solved. During the proof search, the manager follows PUCT [32] selection to traverse from the root to a leaf node. Next, the PCN estimates the cost of the selected leaf node, denoted by v l . v l is a heuristic value representing the log estimated number of positions that must be examined to solve this node. If the value is larger than a designated threshold, i.e. v l ≥ v thr , its proof cost is considered too high to warrant a job. The manager will then continue to follow MCTS, expanding the node and backpropagating v l to the root. Alternatively, if v l < v thr , the leaf node is highly likely to be solved outright by a worker, at which

Fine-Tuning

Solutions

(b) Workers

(a) Manager

PCN

Worker

Jobs

Solved

/Critical

Positions

Models

⋯

Self-Play

(c) Online Fine-Tuning Trainer

𝑣𝑡ℎ𝑟

Queue

e

r

a

b

c

d

f

g

j

k

l

m

n

h

i

MCTS Selection Path

Selection 1: r a g j

Selection 2: r a g k m

Selection 3: r a g k n

Selection 4: r d

Selection 5: r b h

Selection 6: r b i

Solved

Solved

point a job is created. Job granularity is therefore controlled with v thr . Larger v thr generates more difficult jobs with higher failure rates, while smaller v thr leads to easier but more numerous jobs. A balanced v thr should be set according to the game instance and worker capabilities.

Workers are game solving programs that are limited by specific constraints, say, a given time limit. To keep the heuristic consistent during the proof search, workers use the same PCN weights as the manager. If the job is solved within the given constraint, the worker returns the result, either a win or a loss, back to the manager; otherwise, it returns an unknown. Once the manager receives the job result, it updates the search tree accordingly. For unsolved jobs, the manager expands the nodes that generated the corresponding jobs. The interaction between the manager and the workers is shown between Figure 1(a) and 1(b).

3.2 Online Fine-Tuning Trainer

The Online Fine-Tuning Trainer (OFT) maintains the PCN during the proof search so that the manager and workers have access to ever improving heuristics. Without online fine-tuning, both the manager and workers simply use a fixed PCN, denoted by θ 0 , trained via the AlphaZero self-play process. The OFT starts with θ 0 , then fine-tunes the weights via updates during the proof search. To do this, the manager picks out solved and/or critical positions in its search tree, adds them to the list of training samples, then the OFT uses them to perform self-play and training. The fine-tuned PCN ( θ 1 , θ 2 , ..., θ t , ... ) can then be used to further guide the manager and workers towards a faster proof. The manager and workers update to the most recent θ t immediately when a new PCN checkpoint is trained by the OFT. The above iterative process is shown in Figure 1. Details are provided in the following sections.

3.2.1 Online fine-tuning trainer with solved positions

During the proof search, many previously unsolved positions may become solved in the manager's search tree. This new information can be used by the OFT to improve the accuracy of the PCN. Figure 2 provides an example of a manager's AND-OR search tree and six recent selection paths. In this example, positions j and d are marked as solved and sent to the OFT after the first and fourth selection, respectively. The OFT maintains a queue that stores these solved positions, as shown in Figure 1(c). Self-play games are generated as in normal PCN training [14]. However, in the optimization phase, the OFT randomly samples training data not only from the generated self-play games, but also from the queue of solved positions. For these solved positions, the cost values are always set to zero (i.e. solved to be a win, from the perspective of the OR-player), since no nodes need to be examined to solve the position. The OFT only samples 10% of training data from the solved queue during optimization to avoid overfitting, where the remaining 90% are sampled from self-play games. In addition, the queue only stores the most recent 1,000 solved positions received from the manager. During self-play, when using θ t to evaluate positions that are solved by the manager, it is highly likely to predict costs close to zero. From the AND-player's perspective, it favors moves that lead to larger costs to delay the OR-player's victory. Therefore, self-play naturally explores positions which have not yet been solved in the manager's search tree.

3.2.2 Online fine-tuning trainer with critical positions

Other than solved positions, we can also improve the PCN with specific positions of interest chosen from the manager's current search tree. Positions are considered critical if they are selected in the most recent MCTS iterations in the manager. For example, in the first selection in Figure 2, all positions r , a , g , and j in the MCTS selection path are considered critical positions. During self-play, the trainer randomly chooses one critical position, and performs self-play starting from that position. Thus, θ t can provide more accurate predictions for positions that proof search is currently exploring.

A more selective process can be used to improve the quality of critical positions. First, we can omit r , since self-play from r is already performed to train θ 0 . Ideally, we would prefer to focus on deeper unsolved positions. To achieve this, we only consider the leaf position in the selection path as critical. Also, the OFT maintains a queue in which only the recent 1,000 critical positions are stored. This way, the OFT can focus on the most urgent positions which are likely to be solved soon. As these positions are also usually sent to the workers (if the PCN value v ≤ v thr ), the workers can also take advantage of θ t . Next, we can omit leaf positions solved solely by the manager; i.e. leaf nodes that were solved not as jobs. For example, j will not be considered critical in the first selection in Figure 2. Since j is already solved, it is not necessary to perform self-play from that position. Only m , n , h , and i will be sent to the OFT as critical positions in the second, third, fifth, and sixth selection, respectively. Note that r , a , and g were critical positions before b and k became critical, since the parent nodes are always expanded before their children. Thus, the trainer will gradually fine-tune the PCN by focusing only on deeper critical positions to help avoid redundancy during fine-tuning.

In summary, the pre-trained θ 0 learns general heuristics by exploring from empty games, while the online θ t refines its heuristics for specific positions of interest. As a side note, the fine-tuning process is related to the catastrophic forgetting phenomenon [33], as the focus is shifted from one part of the proof search to another. Interestingly, forgetting is not only acceptable in this context, but probably even preferred, because the heuristic only needs to be accurate for the part of the search space the manager is currently working on. Additionally, the two proposed methods are independent and can be combined. We evaluate these methods in our experiments.

3.3 Manager job assignment improvements

Job assignment refers to the manager's responsibility of dividing the overall problem into distinct jobs. Better job assignment schemes can eliminate redundancy and improve parallelism. We propose three techniques to further improve the efficiency of job assignment, which we call virtual solving , topk selection , and AND-player job assignment .

Virtual solving. When a job is assigned to workers, we assume that the job result will be solved, even before it is actually returned by a worker. The virtually solved outcome is backpropagated as a normal job outcome. This technique has similar concepts to the virtual loss [34], virtual win [29], and Young Brothers Wait Concept (YBWC) [35], which were used to avoid repeatedly searching superfluous nodes during the proof search. For example, in Figure 2, assume the manager selects a path from r to a leaf h and assigns the job to a worker, at which point h is immediately marked as virtually solved. Its parent node b , an OR-node, is then also marked as virtually solved. Furthermore, if nodes a , c , and e ( d is solved already) are all solved or virtually solved, their parent r will also be marked as virtually solved. When the job result returns, the manager reverts the virtually solved markers and updates the status of all nodes accordingly. The virtual solving technique can provide a highly efficient job assignment scheme, in that the manager search tree can be nearly the same as the solution tree if most virtually solved nodes are indeed winning.

Topk selection. We exploit the fact that all child nodes must be solved for every AND-node to improve parallelism. At each AND-node, we select uniformly at random among the top k unsolved children that are likely to be sent off as jobs eventually, i.e. those with the top k highest PUCT scores. For example, in Figure 2, assume k = 2 and nodes a and b are the top two children of AND-node r ; nodes k and l are the top two for g . Note that we omit node j because it is already solved. At node r , the manager selects between a and b with equal probability. Note that selections at OR-nodes remain unchanged. In addition, we only apply topk selection when the simulation count of the AND-node is larger than k . Topk selection improves parallelism by allowing the manager to assign more jobs simultaneously, when it is combined with virtual solving. We use k = 4 in our experiments.

AND-player job assignment. We only distribute AND-nodes as jobs, i.e. OR-nodes are never assigned and are directly expanded in the manager. For example, in Figure 2, the OR-node b is not assigned as a job even if v b < v thr . The manager creates the AND-node h from the leaf node b , then assigns it to a worker as a job. The underlying intuition is that assuming the PCN policy head output is accurate as a move ordering heuristic, the first guess will often be the move that leads to a solution for OR-nodes. Therefore, by skipping OR-nodes job assignment entirely, the manager gains a 1-ply look ahead. In practice, all three job assignment schemes are applied simultaneously.

4 Experiments

We demonstrate our online fine-tuning game solver by solving several three-move 7x7 Killall-Go openings. 7x7 Killall-Go is a variant of Go, where the rules are the same except that: (a) Black places two stones initially, and (b) Black wins if all white stones are killed; otherwise, White wins. Since White aims to live, winning specific openings for this variant is equivalent to solving a L&D problem. Many Go experts believe that 7x7 Killall-Go is a win for White. So far, no proof has been published yet. In this paper, we only focus on weakly solved games [36] in which White wins. Thus, White is considered the OR-player throughout.

4.1 The 7x7 Killall-Go solver

We build our 7x7 Killall-Go solver upon an AlphaZero training framework [37]. First, we pre-train a PCN θ 0 [14] to serve as heuristics for the game solver (starting from an empty board). We incorporate the Gumbel AlphaZero algorithm [38] into PCN training, since it performs equivalently well even with a small simulation count. This reduces the computation cost for online fine-tuning without compromising accuracy. The pre-training took around 52 1080Ti GPU-hours. Next, we incorporate several useful techniques into the solver to accelerate solving. This includes relevance zone-based search (RZS) [13], zone pattern tables [39], and GHI handling to deal with cycles in Go [40]. This solver is then used as workers in a distributed game solver. The manager is also based on the above solver, with the job assignment techniques added, as described in subsection 3.3. The OFT is similar to the PCN pre-training, but with fine-tuning as described in subsection 3.2.

Two kinds of distributed game solvers are considered for our experiments. The baseline solver uses the manager and worker only, while using a pre-trained, fixed θ 0 as the heuristic throughout the whole proof search. In contrast, the online fine-tuning solver uses the OFT to fine-tune the PCN heuristic dynamically during the proof search. In addition, we consider three variations of online fine-tuning solvers using solved positions (SP), critical positions (CP), and a combination of both (SP+CP). Both solvers use v thr = 16 . 5 for the manager job granularity. 4 For fairness, we ran both solvers on 9 1080Ti GPUs. The baseline solver uses one GPU for the manager and eight GPUs shared among workers. For the online fine-tuning solver, the manager and trainer each uses one GPU, while workers share the remaining seven GPUs. Detailed implementations and other machine configuration details are specified in the appendix.

B

A

D

2

1

C

1

E

2

F

D

1

C

A

B

1

(b) Knight's move

(K)

1

A

2

1

(c) Diagonal jump

(D)

B D

A

1

1

2

C

E

(d) Stretch (S)

We select a set of three-move openings based on recommendations from experts, including a professional 9-dan player. These openings can be classified into four groups, named after their commonly

shared first move opening: jump (J), knight's move (K), diagonal jump (D), and stretch (S), shown in Figure 3a, 3b, 3c, and 3d respectively. For each opening group, experts also suggest the most likely winning move for White. We split these openings into several three-move openings by exploring Black's possible replies. For each opening group, we select the most difficult replies by Black according to expert recommendations and the PCN policy head output. For simplicity, in the rest of the paper, JA represents the position resulting from Black playing at A in the jump group; KB represents Black playing at B in the knight's move group, etc. The gray solid squares represent moves that are also suggested by the PCN, but cannot be solved by both of the baseline and online fine-tuning solvers in one day. With limited computing resources, we leave these openings for future work. In total, we use 16 three-move openings as shown in Figure 3.

4.2 Baseline versus online fine-tuning

Table 1 lists statistics for solving the 16 three-move openings by the baseline solver and three variants of online fine-tuning solvers. In general, all online fine-tuning solvers outperform the baseline solver in most openings. ONLINE-SP, ONLINE-CP, and ONLINE-SP+CP, require only about 48.53%, 21.69%, and 23.07% of the visited nodes, and 52.67%, 23.54%, and 24.99% of the computing time compared to BASELINE. This shows that fine-tuning PCNs with critical positions, which are currently being solved by either the manager or workers, provides better heuristics for the current search tree and accelerates the solving process. Furthermore, ONLINE-SP+CP has nearly the same performance as ONLINE-CP, with both methods outperforming ONLINE-SP. This means that training with critical positions is more important than solved positions. To reduce the overhead of sending both solved and critical positions, we simply choose ONLINE-CP for further analysis, i.e. all instances of online fine-tuning solver for the rest of this section refers to ONLINE-CP. In conclusion, these results indicate that θ 0 provides less accurate heuristics, which impacts the proof search negatively. By performing online fine-tuning with either solved or critical positions, we can fine-tune the PCN dynamically according to the manager's current focus and therefore find faster solutions.

Table 1 also shows another interesting result: the larger the problem, the better the improvement. For better visualization, the solving times are depicted as a bar chart in Figure 4, where the x-axis is sorted according to the solving time of the baseline solver. In Figure 4, the online fine-tuning solver solves all openings within 40,000 seconds, while the baseline solver uses more than one day to solve six openings. Most impressively, for JA , the online fine-tuning solver performs about 6.35 times faster than the baseline, reducing the computation time from 142,115 to 22,384 seconds, while the number of visited nodes is reduced from 8.96 billion to 1.29 billion or so nodes.

The online fine-tuning solver does not always perform better than the baseline, especially when the openings are relatively easy to solve for the baseline. In Table 1, the baseline uses less time to solve SA and SC . The following reasons may be why this limitation exists for smaller problems. First, the

KC

SA

SC

KA

SB

KE

KD

SE

DA

SD

JA

KF

JB

KB

0

20

,

000

40

,

000

60

,

000

80

,

000

100

,

000

120

,

000

140

,

000

160

,

000

Time (s)

BASELINE

ONLINE-SP

ONLINE-CP

ONLINE-SP+CP

1

1

2

3

4

5

A

B

(a) Sub-position of

JA

.

0

20

40

60

80

100

120

140

160

180

15

20

25

Iteration

Length

(b) Average length of critical positions.

online fine-tuning solver relies on a trainer to fine-tune the PCN. The quicker the problem can be solved, the less time the trainer has to fine-tune specific PCN weights. Consequently, it has a weaker impact on improvement. This is corroborated by the fact that these two openings end up with less than 20 PCN versions, 5 as shown in Table 1. Second, when compared to the baseline, the online fine-tuning solver has less computing power for the workers. The trainer overhead takes up a GPU and leaves workers with seven instead of eight GPUs used by the baseline.

It is also worth mentioning that all three-move openings in both jump and stretch are solved. As these openings are considered the most difficult moves, we expect that both jump and stretch (i.e., the two-move openings) can probably be solved completely in the near future, with more computing resources.

4.3 Behavior analysis for the online fine-tuning solver

We further analyze the behavior of two solvers by studying the opening JA , where the online finetuning solver has the largest speedup among all openings, i.e. 6.35 times faster than the baseline. We observe several positions in which the winning moves for White differed between the two solvers. For example, a crucial sub-position in the solution tree is shown in Figure 5a. The baseline and online fine-tuning solver chose moves A and B to search, respectively. We examine θ 0 and find that the

probabilities from the policy network for moves A and B are 0.416 and 0.133, respectively. As a result, the baseline solver has a lower chance to explore B. In contrast, with the help of the OFT, the online fine-tuning solver quickly realizes that solving B is faster, though it still attempted to search A initially, as it starts with the same θ 0 . In terms of node counts, the baseline spent a total of 1.63 billion nodes, with approximately 1.48 billion (91.00%) and 35.50 million (2.18%) nodes spent on searching A and B, respectively, while the online fine-tuning solver spent a total of 136.47 million nodes on that position, with approximately 22.13 million (16.21%) and 95.17 million (69.74%) nodes spent on searching A and B, respectively. This example clearly demonstrates the advantage of using the OFT.

Next, we investigate the set of critical positions maintained by the online fine-tuning trainer, as described in subsection 3.2.2. During fine-tuning, the trainer randomly selects positions from the queue, then runs self-play games from these positions. Figure 5b shows the average path length of critical positions during training in the y-axis, and the training iteration in the x-axis, where the trainer generates a new PCN version for every iteration. The length starts at around 10 and gradually increases to nearly 25 in the end. This is because as the manager search tree grows through node expansion, the critical positions are chosen from deeper parts of the tree. The curve also fluctuates as the proof search progresses. This is because the manager tends to focus on a sub-problem at a time. When a subtree is solved, the manager may then shift its attention to other unsolved parts of the proof search, which can have a relatively shallower depth. We also analyze similar figures for other openings in the appendix.

4.4 Updating PCNs in online fine-tuning

We investigate the impact of updating PCNs for the manager and workers during online fine-tuning. We select four openings, JC , KE , DA , and SE , one from each opening group, for this experiment. Table 2 summarizes results, where BASELINE denotes the baseline solver, ONLINE-CP denotes the online fine-tuning solver that updates PCNs for both the manager and workers (as described in subsection 3.2.2), ONLINE-CP-M denotes updating the PCN for the manager only, and ONLINE-CP-W for workers only.

First, updating PCNs for both the manager and workers performs the best. By using consistent PCNs, jobs assigned by the manager are efficiently solved by the workers. With inconsistent PCNs, the results can be even worse than the baseline. Generally, ONLINE-CP-M outperforms ONLINE-CP-W, except for the opening SE . We find that the pre-trained PCN causes ONLINE-CP-W to divide its computing resources across several OR nodes (white moves), many of which are relatively difficult to solve. In contrast, for ONLINE-CP-M, with the updated PCNs, the manager focuses on one white move (or a smaller number of white moves), of which they are much easier to solve. Thus, even if there is a mismatch between the manager and workers in ONLINE-CP-M, focusing on one good white move can still result in efficiency.

Table 2: Impact of updating PCNs in the online fine-tuning solver. The number represents the time (in seconds) for solving each opening by using different methods.

Table 3: Ablation study for job assignment schemes in the online fine-tuning solver. The number represents the time (in seconds) for solving each opening by using different methods.

4.5 Ablation study for job assignment schemes

We conduct an ablation study in the online fine-tuning solver to analyze the impact of job assignment schemes, described in subsection 3.3. We only include ablations for topk selection and AND-player job assignment, since virtual solving is required to avoid job redundancy. The ablation study is performed on the same four openings as subsection 4.4, JC , KE , DA , and SE . Table 3 summarizes the ablation results, where ONLINE-CP denotes the online fine-tuning solver that uses both schemes; the other three versions denote the ablations by removing specific schemes from ONLINE-CP. If the topk

selection is removed, the manager always selects the best child during selection for both AND-nodes and OR-nodes. If we do not follow the AND-player job assignment scheme (abbreviated as AND assg. in the table), the manager assigns both AND-player and OR-player jobs.

From Table 3, ONLINE-CP performs the best in general. In particular, ONLINE-CP only requires around 48.37% of the computing time on average over all four openings, compared to the solver without both schemes (the last row in the table). When comparing each technique individually, the improvement varies from problem to problem.

5 Discussion

This paper demonstrates the potential of using online fine-tuning for game solving. On average across multiple openings, our proposed online fine-tuning solver only uses 23.54% of the computation time compared to the baseline. Our distributed game solver is the first online fine-tuning method for problem solving based on AlphaZero-like algorithms. Although we focus on online fine-tuning throughout this paper, we can also claim that the complete distributed game solver is a life-long learning system. The online trainer continuously refines heuristics for unfamiliar (unsolved) positions, and forgets the previously learned knowledge (solved positions). However, this forgotten knowledge is saved (remembered) in the manager's solution tree. As a result, the worker and trainer will not need to evaluate these positions again.

There are many other topics for future investigation. Our experiments on the four challenging 7x7 Killall-Go opening groups show that two groups are likely to be solved in the near future. However, for the other two, or even 7x7 Killall-Go in its entirety, we expect more novel techniques are needed. As for the standard Go game, the largest solved board to date is only 5x6 in size [41], with no published progress in 14 years. We expect online fine-tuning to be one of the key improvements that can help push this boundary. As for generality, our method is not limited to Go but can be easily applied to other two-player zero-sum games like Hex or Othello. Moreover, we expect it has the potential to extend to single-player games such as Rubik's Cube, or even to other non-game fields, such as automated theorem proving [17] or chemical syntheses [42, 43].

Acknowledgments and Disclosure of Funding

This research is partially supported by the National Science and Technology Council (NSTC) of the Republic of China (Taiwan) under Grant Number NSTC 111-2222-E-001-001-MY2, NSTC 111-2221-E-A49-101-MY2, NSTC 110-2221-E-A49-067-MY3, and NSTC 111-2634-F-A49-013.

References

H Jaap van den Herik, Jos W H M Uiterwijk, and Jack Van Rijswijck. Games solved: Now and in the future. Artificial Intelligence , 134(1):277-311, 2002.

Jonathan Schaeffer. One Jump Ahead: Challenging Human Supremacy in Checkers. ICGA Journal , 20(2):93-93, 1997.

Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep Blue. Artificial intelligence , 134(1-2):57-83, 2002.

David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature , 529(7587):484-489, 2016.

Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature , 575(7782):350-354, 2019.

David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature , 550(7676):354-359, 2017.

David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play. Science , 362(6419):1140-1144, 2018.

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, Timothy Lillicrap, and David Silver. Mastering Atari, Go, chess and shogi by planning with a learned model. Nature , 588(7839):604-609, 2020.

Adam Gleave, Michael Dennis, Cody Wild, Neel Kant, Sergey Levine, and Stuart Russell. Adversarial Policies: Attacking Deep Reinforcement Learning. In 8th International Conference on Learning Representations, ICLR 2020 , 2020.

Li-Cheng Lan, Huan Zhang, Ti-Rong Wu, Meng-Yu Tsai, I-Chen Wu, and Cho-Jui Hsieh. Are AlphaZero-like Agents Robust to Adversarial Perturbations? In 36th Advances in Neural Information Processing Systems, NIPS 2022 , 2022.

Jonathan Schaeffer, Neil Burch, Yngvi Björnsson, Akihiro Kishimoto, Martin Müller, Robert Lake, Paul Lu, and Steve Sutphen. Checkers Is Solved. Science , 317(5844):1518-1522, 2007.

Chao Gao, Martin Müller, and Ryan Hayward. Focused Depth-first Proof Number Search using Convolutional Neural Networks for the Game of Hex. In 26th International Joint Conference on Artificial Intelligence, IJCAI 2017 , volume 17, pages 3668-3674, 2017.

Chung-Chin Shih, Ti-Rong Wu, Ting Han Wei, and I-Chen Wu. A Novel Approach to Solving Goal-Achieving Problems for Board Games. In 36th AAAI Conference on Artificial Intelligence, AAAI 2022 , volume 36, pages 10362-10369, 2022.

Ti-Rong Wu, Chung-Chin Shih, Ting Han Wei, Meng-Yu Tsai, Wei-Yuan Hsu, and I-Chen Wu. AlphaZero-based Proof Cost Network to Aid Game Solving. In 10th International Conference on Learning Representations, ICLR 2022 , 2022.

Stephen McAleer, Forest Agostinelli, Alexander Shmakov, and Pierre Baldi. Solving the Rubik's Cube with Approximate Policy Iteration. In 7th International Conference on Learning Representations, ICLR 2019 , 2019.

Forest Agostinelli, Stephen McAleer, Alexander Shmakov, and Pierre Baldi. Solving the Rubik's cube with deep reinforcement learning and search. Nature Machine Intelligence , 1(8):356-363, 2019.

Guillaume Lample, Marie-Anne Lachaux, Thibaut Lavril, Xavier Martinet, Amaury Hayat, Gabriel Ebner, Aurélien Rodriguez, and Timothée Lacroix. HyperTree Proof Search for Neural Theorem Proving. In 36th Advances in Neural Information Processing Systems, NIPS 2022 , 2022.

David Wu. Deep-Learning the Hardest Go Problem in the World. https://blog. janestreet.com/deep-learning-the-hardest-go-problem-in-the-world/ , 2019. Accessed: 2023-01-19.

Wim Pijls and Arie de Bruin. Game tree algorithms and solution trees. Theoretical computer science , 252(1-2):197-215, 2001.

George C. Stockman. A minimax algorithm better than alpha-beta? Artificial Intelligence , 12 (2):179-196, 1979.

Louis Victor Allis, Maarten van der Meulen, and H Jaap van den Herik. Proof-number search. Artificial Intelligence , 66(1):91-124, 1994.

Mark H M Winands, Yngvi Björnsson, and Jahn-Takeshi Saito. Monte-carlo tree search solver. In International Conference on Computers and Games , pages 25-36. Springer, 2008.

Tristan Cazenave and Abdallah Saffidine. Score Bounded Monte-Carlo Tree Search. In 7th International Conference on Computers and Games, CG 2010 , pages 93-104. Springer Berlin Heidelberg, 2011.

Timo V Ewalds. Playing and Solving Havannah. Master's thesis, University of Alberta, 2012.

Ting Han Wei, Chao-Chin Liang, I Wu, Lung-Pin Chen, et al. Software development architecture for job-level algorithms. ICGA Journal , 38(3):131-148, 2015.

Michael Bowling, Neil Burch, Michael Johanson, and Oskari Tammelin. Heads-up limit hold'em poker is solved. Science , 347(6218):145-149, 2015.

Abdallah Saffidine, Nicolas Jouandeau, and Tristan Cazenave. Solving BRREAKTHROUGH with Race Patterns and Job-Level Proof Number Search. In 13th International Conference on Advances in Computer Games, ACG 2011 , pages 196-207. Springer, 2011.

Philip Henderson, Broderick Arneson, and Ryan B Hayward. Solving 8x8 Hex. In 21st International Joint Conference on Artificial Intelligence, IJCAI 2009 , volume 9, pages 505-510, 2009.

I-Chen Wu, Hung-Hsuan Lin, Der-Johng Sun, Kuo-Yuan Kao, Ping-Hung Lin, Yi-Chih Chan, and Po-Ting Chen. Job-Level Proof Number Search. IEEE Transactions on Computational Intelligence and AI in Games , 5(1):44-56, 2012.

Ayumu Nagai. Df-pn Algorithm for Searching AND/OR Trees and Its Applications . PhD thesis, University of Tokyo, Tokyo, Japan, 2002.

Jr-Chang Chen, I-Chen Wu, Wen-Jie Tseng, Bo-Han Lin, and Chia-Hui Chang. Job-level alpha-beta search. IEEE Transactions on Computational Intelligence and AI in Games , 7(1): 28-38, 2014.

Christopher D Rosin. Multi-armed bandits with episode context. Annals of Mathematics and Artificial Intelligence , 61(3):203-230, 2011.

Michael McCloskey and Neal J Cohen. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. In Psychology of learning and motivation , volume 24, pages 109-165. Academic Press, 1989.

Guillaume MJ-B Chaslot, Mark HM Winands, and H. Jaap van den Herik. Parallel Monte-Carlo Tree Search. In 6th International Conference on Computers and Games, CG 2008 , pages 60-71. Springer, 2008.

Rainer Feldmann, Burkhard Monien, Peter Mysliwietz, and Oliver Vornberger. Distributed game-tree search. ICCA Journal , 12(2):65-73, 1989.

Louis Victor Allis. Searching for Solutions in Games and Artificial Intelligence . PhD thesis, Transnational University Limburg, Maastricht, Netherlands, 1994.

Ti-Rong Wu, Hung Guei, Po-Wei Huang, Pei-Chiun Peng, Ting Han Wei, Chung-Chin Shih, and Yun-Jui Tsai. Minizero: Comparative analysis of Alphazero and Muzero on Go, Othello, and Atari games. arXiv preprint arXiv:2310.11305 , 2023.

Ivo Danihelka, Arthur Guez, Julian Schrittwieser, and David Silver. Policy improvement by planning with Gumbel. In 10th International Conference on Learning Representations, ICLR 2022 , 2022.

Chung-Chin Shih, Ting Han Wei, Ti-Rong Wu, and I-Chen Wu. A Local-Pattern Related Look-Up Table. IEEE Transactions on Games , 2023.

Akihiro Kishimoto and Martin Müller. A General Solution to the Graph History Interaction Problem. In 19th AAAI Conference on Artificial Intelligence, AAAI 2004 , volume 4, pages 644-649, 2004.

Erik C D van der Werf and Mark H M Winands. Solving go for Rectangular Boards. ICGA Journal , 32(2):77-88, 2009.

Marwin HS Segler, Mike Preuss, and Mark P Waller. Planning chemical syntheses with deep neural networks and symbolic AI. Nature , 555(7698):604-610, 2018.

Akihiro Kishimoto, Beat Buesser, Bei Chen, and Adi Botea. Depth-First Proof-Number Search with Heuristic Edge Cost and Application to Chemical Synthesis Planning. In 33rd Advances in Neural Information Processing Systems, NIPS 2019 , pages 7224-7234, 2019.

Andrew J Palay. Searching with Probabilities . PhD thesis, Carnegie Mellon University, 1983.

Akihiro Kishimoto and Martin Müller. A solution to the GHI problem for depth-first proofnumber search. Information Sciences , 175(4):296-314, 2005.

A Implementation details

A.1 PCN training

We basically follow the same PCN training method by Wu et al. [14] but replace the AlphaZero algorithm with the Gumbel AlphaZero algorithm [38], where the simulation count is set to 32 6 in self-play and starts by sampling 16 actions. The architecture of the PCN contains three residual blocks with 256 hidden channels. A total of 400,000 self-play games are generated for the whole training. During optimization, the learning rate is fixed at 0.02, and the batch size is set to 1,024. The PCN is optimized for 500 steps for every 2,000 self-play games. The pre-trained PCN requires around 13 hours to train on a machine with four 1080Ti GPUs, i.e. 52 1080Ti GPU-hours. For the online trainer, we use the same hyperparameters as the pre-trained PCN but only use one GPU.

A.2 7x7 Killall-Go solver

Our solver is built upon the state-of-the-art (SOTA) 7x7 Killall-Go solver [13] except for the following three changes. First, our solver uses PCN as heuristics while the SOTA solver trains a network with Faster to Life (FTL) techniques. Both networks aim to provide a faster move for solving, but FTL requires additional ( komi 7 ) settings in solving, so PCN is much easier to use in our solver. Second, we implement the transposition table based on Shih et al. [39]. This greatly reduces the solving time. Finally, we implement a solution for resolving Graph-History-Interaction (GHI, i.e. cycles in Go) [44] problems to ensure the correctness of reusing solutions in the transposition table, based on Kishimoto and Müller [40, 45]'s GHI solution.

A.3 Worker design

The worker is itself a Killall-Go solver. It is GPU bound, i.e. it relies on GPUs more than CPUs since the PCN (a neural network) requires intensive GPU computation. Thus, to fully utilize GPU resources, we implement batch GPU inferencing to accelerate PCN evaluations for workers. In practice, we collect 48 workers together in one process with multiple threads. The process runs MCTS selection for each worker independently. Namely, a total of 48 leaf nodes are generated and evaluated by PCN with one GPU at once. The 48 leaf nodes are collected as a batch for batch GPU inferencing, with a batch size of 48. This method greatly reduces the solving time when more workers are used. The baseline distributed game solver creates eight processes as workers, each with one GPU, for a total of 384 workers (eight processes with 48 workers). The online fine-tuning solver has the same number of workers for fairness, but uses seven GPUs (one GPU is spared for the online trainer); the configuration is six processes with 55 workers and one process with 54 workers.

B Experiment details

B.1 Setup

All experiments are conducted in three machines, each equipped with two Intel Xeon E5-2678 v3 CPUs, 192G RAM, and four GTX 1080Ti GPUs. We list other hyperparameters in Table 4.

For the memory used in solving, the manager requires 20G RAM for expanding every 1M nodes, and every 48 workers together in one process requires 30G RAM at most. Note that workers use the same amount of memory regardless of problem size. They are limited to 100,000 nodes per job; the job result is 'unsolved' if a solution is not obtained within that limit.

Specifically, for BASELINE with 384 workers, solving KA used 2,103 seconds, required 3G RAM for the manager and 240G RAM for the workers; solving KB used 156,583 seconds, required 170G RAM for the manager and 240G RAM for the workers. However, for BASELINE with only 48 workers, solving KA used 12,151 seconds but only required 2G RAM for the manager and 30G RAM for the workers. Overall, the settings can be varied depending on available machines.

B.2 Scalability of the distributed game solver

To evaluate the scalability of the distributed game solver, we run BASELINE with different numbers of workers on KA . Specifically, the solvers use 384, 192, 96, and 48 workers, using 8, 4, 2, and 1 GPU, respectively. Every 48 workers share one GPU. The results are shown in Table 5. Overall, the speedup is around 1.8 times faster when the number of workers is doubled (up to 384 workers due to our machine limitation).

B.3 Statistics of solving 7x7 Killall-Go three-move openings

Figure 6 shows the next winning moves (the fourth moves) of 16 three-move openings for both baseline and ONLINE-CP solvers. Generally, both solvers solve the openings at the same next moves, except JB . The full solution trees for each opening can be found in this link: https://rlg.iis. sinica.edu.tw/papers/neurips2023-online-fine-tuning-solver/solution-trees . We also provide a tool and a README file for explaining the solution tree.

It is worth mentioning that JA and JB are similar to one of the common josekis 8 played in 19x19 Go. The joseki usually occurs when Black makes a corner enclosure move, also known as shimari in Japanese, like the two stones marked as '1' in JA and JB . Then, White attempts to invade Black's territories by playing at the stone marked as '2'. Judging by the online fine-tuning solver's ability to solve JA and JB , we foresee a high potential to extend our work to solving other 19x19 Go corner josekis in the future.

In addition, Figure 7 shows the curve for average critical position lengths. These curves are all similar in the sense that it starts with small average lengths, which gradually increases during fine-tuning.

Table 6, Table 7, Table 8 and Table 9 list the experiment results of the baseline and three variants of online fine-tuning solvers respectively, in more detail than those in Table 1 in the main text. These tables include the number of nodes for solving, the solving time in seconds, the number of nodes used in the manager, the number of jobs, the average time for solving each job, the average number of nodes for solving each job, the number of updated PCNs, the success rate of solving jobs, and the average worker load during solving. In general, the solving time is correlated with the number of nodes and the number of jobs. For online fine-tuning, the solving time is also correlated with the number of PCNs as the trainer updates PCNs at a stable speed. Note that the number of PCNs is always 0 for the baseline solver, as they do not update PCNs during solving.

1

1

2

3

A

(a) JA

1

1

2

3

A

B

(b) JB

1

1

2

3

A

1

1

2

3

A

1

1

2

3

A

(e) KA

1

1

2

3

A

1

1

2

3

A

(g) KC

1

1

2

3

A

1

1

2

3

A

(i) KE

1

1

2

3

A

(j) KF

1

2

1

3

A

(k) DA

1

1

2

3

A

(l) SA

1

1

2

3

A

(m) SB

1

1

2

3

A

(n) SC

1

1

2

3

A

(o) SD

1

1

2

3

A

(p) SE

In our experiments, the average success rates of solving jobs are around 97.30%, 98.44%, 99.14% and 99.08% for the baseline and the online fine-tuning solvers, respectively. In addition, for some quickly solved openings, e.g. KC , SA , and SB , the average time for solving each job is far less than other difficult openings. While the workers are able to solve jobs quickly, the managers are relatively unable to create enough jobs for the workers, causing the workers to be relatively idle (lower avg. worker loading). Compared with the baseline solver, online fine-tuning solvers have better success rates of solving as well as lesser nodes for each job. This confirms that online fine-tuning successfully fine-tuned the PCNs for critical positions that the manager is interested in, thereby increasing the job efficiency overall.

B.4 Different PCN thresholds

Weexamine different v thr from 11 . 5 to 21 . 5 on opening JC , using the baseline solver. The experiment result is presented in Table 10, where the four columns represent the examined v thr , the total solving time, the average time for workers to solve jobs, and the job success rate. Among these PCN thresholds, we consider v thr = 16 . 5 to be a balanced setting as it performs well in the three metrics. However, the results also show that the performance is not necessarily sensitive to different v thr settings, i.e. the solving time is similar when v thr ∈ (15 . 5 , 17 . 5) .

0

20

40

60

80

100

120

140

160

180

15

20

25

Iteration

Length

JA

(a) JA

0

20

40

60

80

100

120

140

160

180

200

220

240

260

15

20

25

30

35

Iteration

Length

JB

(b) JB

0

5

10

15

20

25

30

35

40

45

50

55

60

12

14

16

18

20

22

24

Iteration

Length

0

10

20

30

40

50

60

70

80

90

100

12

14

16

18

20

Iteration

Length

2

4

6

8

10

12

14

16

18

12

14

16

18

20

Iteration

Length

KA

(e) KA

0

50

100

150

200

250

300

350

12

14

16

18

20

22

24

Iteration

Length

KB

1

1

.

5

2

2

.

5

3

3

.

5

4

4

.

5

5

5

.

5

6

6

.

5

7

11

12

13

14

15

Iteration

Length

KC

(g) KC

0

20

40

60

80

100

120

140

15

20

25

Iteration

Length

KD

2

4

6

8

10

12

14

16

18

20

22

24

26

28

30

32

12

14

16

18

20

22

Iteration

Length

KE

(i) KE

0

20

40

60

80

100

120

140

160

180

200

220

240

260

280

15

20

25

Iteration

Length

KF

(j) KF

As demonstrated in the table, v thr outside of this range deteriorates the solving performance. On the one hand, when v thr is too high, e.g. v thr = 21 . 5 , only about 95% of jobs can be solved, implying that about 5% of the jobs are wasted. On the other hand, when v thr is too low, e.g. v thr = 11 . 5 , the

0

20

40

60

80

100

120

140

160

180

200

220

240

260

15

20

25

Iteration

Length

DA

(k) DA

1

2

3

4

5

6

7

8

9

10

15

20

25

Iteration

Length

SA

(l) SA

1

2

3

4

5

6

7

8

9

10

11

12

13

12

14

16

Iteration

Length

SB

(m) SB

2

4

6

8

10

12

14

16

12

14

16

Iteration

Length

SC

(n) SC

0

20

40

60

80

100

120

140

160

180

12

14

16

18

20

22

Iteration

Length

SD

(o) SD

0

10

20

30

40

50

60

70

80

90

100

15

20

25

Iteration

Length

SE

(p) SE

assigned jobs can be solved quickly with a high success rate. However, this requires the manager to assign more jobs, which increases the overhead of handling job assignments between the manager and the workers, thereby increasing the solving time. Note that the appropriate v thr may vary for different games and for different numbers of available workers. It is possible to adjust v thr dynamically during solving, which is left for future work.

B.5 Comparison to offline fine-tuning

We now investigate how much benefit we can gain from offline fine-tuning for a specific opening. To do this, we first train θ 0 by generating 400,000 self-play games (around 52 1080Ti GPU-hours) from the empty board. The resulting network is the same as the one referred to as θ 0 in the main text. Next, we fine-tune θ 0 by generating 200,000 additional self-play games (around 26 1080Ti GPU-hours) from the specific opening we are interested in. That is, if we want to solve the opening JC , we generate self-play games starting from that opening, and perform updates on θ 0 to obtain what we refer to as θ ′ 0 -JC . For this experiment, we used four openings, so the networks θ ′ 0 -JC , θ ′ 0 -KE , θ ′ 0 -DA , and θ ′ 0 -SE were produced. Lastly, in the baseline case, we do not update the network with critical positions; the same network is used all throughout the proof search. In ONLINE-CP, critical positions are chosen and the θ ′ 0 is further fine-tuned using the OFT (resulting in θ ′ 1 , θ ′ 2 , ..., θ ′ t , ...).

Table 11 shows the times for solving these four openings with and without offline fine-tuning. The left two columns use θ 0 while the right two columns use θ ′ 0 . With offline fine-tuning, the solving times for these openings generally decrease in the baseline solver, since the θ ′ 0 is specifically fine-tuned for each opening, but exceptions may still occur, as in opening JC . However, when using θ ′ 0 , the solving times for ONLINE-CP increase for opening JC , DA , and SE . This may be because θ ′ 0 only helps learn better heuristics for the opening positions, but does not always guarantee providing accurate heuristics for all varieties of positions during solving. In addition, it is worth noting that although

offline fine-tuned θ ′ 0 accelerates the solving time for the baseline solver, it is impractical since we cannot expect to pre-train θ ′ 0 for each opening, especially if our eventual goal is to solve complete games from an empty board outright. In contrast, our online fine-tuning solver provides an automatic method that fine-tunes the PCN dynamically without too much extra computation cost.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper aims to address the game solving problem: providing a game theoretic value to all states in a game. To address the task the paper extends AlphaGo to the game solving case using an approach that distributes game playing scenarios to solvers on parts of the full game tree. Building off prior distributed game solvers, the new technique uses a self-play learning algorithm to improve estimates of the difficulty of solving new game states during the game solving process (as opposed to using a pretrained and frozen estimation method). The work distribution manager incorporates several new heuristics to select the most promising nodes to solve next and avoid wasted computation. These improvements reduce computation time in 7x7 Killall-Go to roughly 1/4 of a non-learning baseline algorithm. Ablations show the heuristics chosen individually and jointly improve the overall solver performance.\n\nSTRENGTHS:\n## originality\n\nModest. The core architecture of distributed game solving and the core AlphaGo algorithm are both established in the literature. The originality of the paper stems from devising a way to do online learning during AlphaGo distributed solving in such a way that state value estimates remain useful throughout the search estimation process. This is not trivial, but is necessarily a highly targeted application.\n\n## quality\n\nGood. The results show substantial improvements over baselines and the ablations are thorough.\n\n## clarity\n\nGood. The substantial background that is needed to understand both distributed game solving and AlphaGo is clearly introduced. Reasoning for each of the major heuristics is clear and the results are unpacked well for an audience not familiar with the target domain.\n\n## significance\n\nModest. Game solving is a somewhat niche topic, so the audience will be limited by that reach. The work itself is building on two prior methods - the improvements are clear, but are sufficiently complex that it is not immediately obvious they will generalize to other games or applications. That said, reducing to 1/4 the computation needed on a fair benchmark is a substantial improvement!\n\nWEAKNESSES:\nOne weakness (hard to address) is the results are only in terms of solving a single game. The game playing literature is awash with game benchmarks and has created a norm of multi-game evaluation to demonstrate generality. This may not apply to game solving, but the lack of multi-game evaluation makes it hard to tell how general the algorithms are. Are there any easy alternative scenarios (like the cited Hex or Rubik's Cube) that would show the algorithm can be generalized to other scenarios? These would be an opportunity to compare against other established baselines, perhaps on smaller or simpler problems.\n\nBelow are detailed suggestions around scalability. The paper would benefit from addressing those concerns as it helps show the potential of the technique over the long run or for larger problems (even if the paper does not directly measure those cases).\n\nThe lifelong learning claim would benefit from direct evidence that the manager remembers what the trainer forgets. See the questions below for more detail.\n\nQUESTIONS:\n- [Q1] How do the core algorithms scale?\n\t- How does performance change with more or fewer workers?\n\t- What are the demands in compute and memory? How do they scale with the size of the problem and/or workers?\n- [Q2] Does critical position prioritization break down when MCTS thrashes between expanding new shallow node and deeper nodes in the tree?\n\t- Often tree search can suffer from a thrash between vastly different branches on the shallow parts of the tree. At a glance this would seem to be a more substantial problem for game solving as the full tree must be solved.\n\t- Is this not a problem in practice? Or is there a way to quantify this effect and measure it's impact?\n- [Q3] Table 1\n\t- What is going on with SB?\n\t- Online-SP and online-CP both do very well compared to the baseline. But online SP+CP does substantially worse than either alone.\n\t- I don't have sufficient knowledge of Go (let alone Killall-Go) to tell why this might be expected or how to evaluate the case. It is interesting as the one scenario that defies the general patterns.\n- [Q4] Figure 4\n\t- Include all 4 variants in the bar plot, not only baseline and online-CP. Also, consider changing the color scheme to be colorblind friendly.\n- [Q5] What is the direct evidence that the manager remembers what the OLT trainer forgets?\n\t- Is there any way to probe the checkpoints to demonstrate this phenomenon?\n\t- The claim is intuitive, but the results would benefit from a direct test / evidence for this claim.\n\n\nBelow are some other less important questions / suggestions:\n- For PCN state estimates:\n\t- How effective is a naive heuristic that computes a similarity between new positions and a database of solved positions? (for solved positions using OLT)\n\t- This would be an alternative to the OLT / PCN approach. I'm not deeply familiar with the game solving literature and so am not sure if this is already a common / established practice.\n- Could the fixed PCN be used to generate compressed/embedded state vectors to improve this matching?\n\t- Could that apply to the learning process to prioritize selection of training samples?\n\nLIMITATIONS:\nThe paper does not address limitations nor negative societal impact. Limitations merit some discussion, specifically around generalization / applicability to other game solving scenarios, classes of games solved (ex: turn-based, 2 player, ...), or computational requirements.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a parallel setup for solving games, which includes online fine-tuning of trained proof cost networks to improve their estimations of the proof cost of nodes specifically for subsets of the state space that the prover is currently focusing on. Experiments show significant reductions in the number of nodes and computation time required to solve numerous 7x7 Killall-Go openings, with improvements also appearing to scale well with problem difficulty.\n\nSTRENGTHS:\n- Relatively straightforward and simple, but good idea.\n- Good empirical results.\n- Paper very well written, easy to follow.\n\nWEAKNESSES:\n- Game solving is a fairly niche topic that probably is mostly of real interest to a relatively small subset of the NeurIPS community (but I think it's fine, not a reason for rejection).\n- The main contribution (online fine-tuning) seems relatively limited in novelty. As far as I'm aware it's novel specifically within the game solving setting, but outside of that, online fine-tuning is not a groundbreaking idea.\n\nQUESTIONS:\n### Suggestions\n- \"Game solving is a much higher challenge than game playing\" --> this is slightly strange phrasing, in particular the \"higher\". Something like \"Game solving is a more difficult challenge than [...]\" would seem more natural.\n\n### Questions\n- \"A two-player zero-sum game is considered solved if there exists a winning strategy for a player which guarantees a winning outcome.\" --> as far as I'm aware this is not correct. A game is solved if we know the outcome under perfect play (either from any state or from the initial state). It may be that this outcome is a draw (or a loss) for the first player though, it need not necessarily be a winning outcome with a winning strategy. I would appreciate a clarification to this question, especially because I also raised this comment when reviewing the same paper submitted to a previous comment, and at the time the authors said the manuscript would be revised accordingly. Which evidently did not happen. If I'm wrong here that's fine, but then please clarify.\n\nLIMITATIONS:\nNo unaddressed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of computing provably optimal solutions to game instances. A naive game solving algorithm must therefore address all the possible actions an opponent may pick before emitting a  judgement about the game instance value. A leading paradigm when developing this category of algorithms is that of employing strong agents to evaluate game states and therefore inform heuristics that aid in exploring the game tree.\n\nThe paper focuses on developing a AlphaZero-based heuristics for game-solving. The main issues are that a self-play learning algorithm is a very robust player on path, but its values do not distinguish among actions leading to games of different lengths (useful in order to solve subgames while exploring less) and they lack prediction quality when considering positions that it would not normally reach in game (and that instead are reached as part of the game solving proof algorithm).\n\nTo address those issues, the paper proposes:\n- the training of Proof Cost Networks (PCN) via the AlphaZero self-play process in order to better inform the expansion heuristics about the complexity of solving a subgame. This is in contrast to naively use a value network that hints at whether the subgame is possibly lost/won, but does not include any complexity estimation.\n- the fine-tuning of the PCN used using selected subgames which have been encountered during the proof process. \n\t- When using solved positions (ie. solved subgames) to fine-tune, those subgames's proof complexity is set to 0 (since it is already proven their value). Then a fine-tuning batch mixing both solved games and self-play games is employed to update the network. This allows to push the heuristics towards game instances which have already been solved, thus efficiently reusing already computed subgame proofs\n\t- When using critical positions (ie. recently explored subgames) to fine tune, the fine-tuning batch is updated by adding experience sampled from applying self-play starting from those positions. This allows to improve PCN's performances on the nodes that will probably be expanded in the future\n\nSTRENGTHS:\n* Clear and exhaustive experimental evaluation\n* Strong experiment results\n* Clear explanation of the heuristics introduced\n\nWEAKNESSES:\n* Purely experimental work applied to a specific subset of go instances. This weakens the generality of the approach because the same heuristics may not apply to other game instances\n* Many dependencies on previous work. Those weaken the overall clarity of the paper for people who are not familiar with the past literature\n* Misleading nomenclature. Online Learning is a terminology which is misleading given the actual technique employed. In particular, Online Learning in games usually refers to *Online convex optimization* techniques, while the paper refers to a Continual Learning technique.  My suggestion would be to switch to Online Fine-tuning to avoid the strong unintended overlap in terminology.\n\nQUESTIONS:\nThe following questions highlight parts of the paper which felt unclear or possibly benefitting from extra detail.\n1. It is not clear how the PCN is trained and its relation to AlphaZero training process. My high-level intuition from reading [14] is that the MCTS algorithm on top of which AlphaZero is based is modified in the Q-values used in the exploration of the game tree. In particular, $\\bar n(s)$ and $\\bar m(s)$ quantities are estimated from episodes of sampled experience (approximating in a \"importance weighting\"-like fashion) and then used to direct self-play towards smaller areas of the tree. My opinion is that including extra details in this direction in Section 2.3 could help readers to have an algorithmic understanding of what it means to the *sampling of self-play strategy* which is employed throughout Section 3.\n2. When using solved/critical positions as heuristics, are the self-play games re-sampled at the moment, or taken from some buffer of experience used to train the first version of the PCN?\n3. Any idea on the theoretical lower bound achievable by having a perfect heuristcs? I.e. is it possible to evaluate the empirical increase in quality of the PCN-based heuristics in terms of the gap between baseline and a perfect heuristic?\n4. How different are the performances of alpha-beta search or PNS algorithms for the problem of game solving?\nOn top of this clarifications, I'd like to hear the authors' opinion regarding the weaknesses I've highlighted in the previous section.\n\nLIMITATIONS:\nThe only limitation of the work is the focus on Go instances and MCTS routines, and those limitations are clearly expressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an AlphaZero-based MCTS search procedure modified for game solving (vs the original game playing objective). The algorithm consists of a carefully-engineered distributed MCTS procedure leveraging GPU-based learned proof-cost-network estimates to grow the search tree in promising directions. The key idea is to interleave search steps with updating the PCN network, on a non-stationary distribution of states encountered during the search, via self-play. The main contributions are algorithmic and empirical. The experiments are conducted on a variant of Go. The results indicate the proposed method reduces the search space and overall computation time on the challenging task of solving 7x7 Killall-Go.\n\nUPDATE: I thank the authors for their detailed response. After reading the other reviews and comments, I'm now more inclined to recommend acceptance and have adjusted my review accordingly.\n\nSTRENGTHS:\n+ The paper tackles a challenging problem of game solving, which has implications for combinatorial search problems.\n\n+ The approach of adjusting the distribution of states explored during the self-play learning step towards those likely to be encountered soon by the outer MCTS search is intuitively clear.\n\n+ The proposed approach consisting of a number of algorithmic choices (distributed search, guided self-play, PCN threshold, manager vs worker roles, etc.) to improve search efficiency. Despite being somewhat heuristic and perhaps not entirely novel, each implementation detail is well explained, intuitively clear and fits nicely into the final algorithm. Overall, the approach seems to be a well-engineered solution to efficiently apply modern GPU-based search procedures to the challenging problem of game solving.\n\n+ The paper includes a detailed experimental investigation. The results indicate that the proposed method clearly has better empirical performance. The utility of training on critical positions is demonstrated clearly. While the baselines could be stronger and other improvements could be made (discussed below), overall, the empirical section is convincing that the proposed method does well on a challenging combinatorial search problem.\n\nWEAKNESSES:\n- The paper does not clearly formalize the online learning objective and does not include a mathematical analysis of the algorithm's performance (e.g., wrt regret).\n\n- The overall approach seems novel but I'm not sure which components of the algorithm are novel. The paper could do a better job of placing its contributions within the larger body of prior work on game solving. In its current form, it's a bit difficult to assess novelty.\n\n- It's difficult to assess how much this approach moves the needle on solving 7x7 Killall-Go or other games compared to prior efforts. Additional baselines and / or domains would help here but likely at large computational expense.\n\n- The paper could go into more detail analyzing the role of the critical node queue, given its importance to the overall performance. On a related note, the paper could do a better job of analyzing the overall stability of the updates to the PCN parameters / model (perhaps in the appendix).\n\n- Code hasn't been included as far as I can tell. The omission will make reproducibility challenging and likely to limit the impact of the paper.\n\n- (minor) The description of the experimental results could be tightened a bit more. Examples with suggestions below.\n  - L283: Comparing \"within 40000 seconds\" with \"more than one day\" could be \"~10 hours\", \"24 hours\" or 2.4x faster. \n  - L286: \"1.29 billion or so\" could be \"~1.29 billion\".\n\nQUESTIONS:\n- Could you please describe the parts of the proposed algorithm or technical ideas that are novel?\n\n- The baseline used is the algorithm without online learning. Are there other baselines for 7x7 Killall-Go from prior work?\n\n- I assume that the reported runtimes include the time spent on updating the parameters. Please confirm this is the case.\n\n- I was a bit confused by L127-L127. The criteria for spawning a job (when $v_l < v_{thr}$) seems to control the distribution of the tree nodes sent to the solver, using the node's $v_{thr}$ value. Thus, increasing v_thr would allow more jobs to be spawned, with the newly included nodes likely being harder to solve. My question is about decreasing v_thr. I'd expect this to generate **fewer** solver jobs, which are likely easier to solve. But L128 claims otherwise, \"smaller v_thr leads to easier but **more numerous** jobs\". What am I missing?\n\n- Can you provide more details about the role of the critical queue? For example, are the nodes in it shallow or deep? How does node depth correlate with sampling frequency / time spent in the queue? How does the PCN network loss curve correlate with the contents of the queue? Additional empirical details, discussion and insight on this subject would be interesting.\n\n- Is it possible to include additional domains? Can you provide information on whether this approach works in other domains?\n\n- Is there a reason code wasn't included? Can it be included?\n\nLIMITATIONS:\nTo an extent. The paper does a better job explaining its strengths than its weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}