SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Two kinds of distributed game solvers are considered for our experiments. The baseline solver uses the manager and worker only, while using a pre-trained, fixed  $\theta_0$  as the heuristic throughout the whole proof search. In contrast, the online fine-tuning solver uses the OFT to fine-tune the PCN heuristic dynamically during the proof search. In addition, we consider three variations of online fine-tuning solvers using solved positions (SP), critical positions (CP), and a combination of both (SP+CP). Both solvers use  $v_{thr} = 16.5$  for the manager job granularity. For fairness, we ran both solvers on 9 1080Ti GPUs. The baseline solver uses one GPU for the manager and eight GPUs shared among workers. For the online fine-tuning solver, the manager and trainer each uses one GPU, while workers share the remaining seven GPUs. Detailed implementations and other machine configuration details are specified in the appendix.
(a) Jump (J)
(b) Knight's move (K)
Figure 3: Four 7x7 Killall-Go opening groups, including (a) four openings, JA-JD; (b) six openings, KA-KF; (c) one opening, DA; (d) five openings, SA-SE.
(c) Diagonal jump (D)
(d) Stretch (S)
We select a set of three-move openings based on recommendations from experts, including a professional 9-dan player. These openings can be classified into four groups, named after their commonly
<sup>4</sup>We choose  $v_{thr} = 16.5$  according to the experiments on different PCN thresholds, as shown in the appendix.
shared first move opening: jump (J), knight's move (K), diagonal jump (D), and stretch (S), shown in Figure 3a, 3b, 3c, and 3d respectively. For each opening group, experts also suggest the most likely winning move for White. We split these openings into several three-move openings by exploring Black's possible replies. For each opening group, we select the most difficult replies by Black according to expert recommendations and the PCN policy head output. For simplicity, in the rest of the paper,  $JA$  represents the position resulting from Black playing at A in the jump group;  $KB$  represents Black playing at B in the knight's move group, etc. The gray solid squares represent moves that are also suggested by the PCN, but cannot be solved by both of the baseline and online fine-tuning solvers in one day. With limited computing resources, we leave these openings for future work. In total, we use 16 three-move openings as shown in Figure 3.
4.2 Baseline versus online fine-tuning
Table 1 lists statistics for solving the 16 three-move openings by the baseline solver and three variants of online fine-tuning solvers. In general, all online fine-tuning solvers outperform the baseline solver in most openings. ONLINE-SP, ONLINE-CP, and ONLINE-SP+CP, require only about  $48.53\%$ ,  $21.69\%$ , and  $23.07\%$  of the visited nodes, and  $52.67\%$ ,  $23.54\%$ , and  $24.99\%$  of the computing time compared to BASELINE. This shows that fine-tuning PCNs with critical positions, which are currently being solved by either the manager or workers, provides better heuristics for the current search tree and accelerates the solving process. Furthermore, ONLINE-SP+CP has nearly the same performance as ONLINE-CP, with both methods outperforming ONLINE-SP. This means that training with critical positions is more important than solved positions. To reduce the overhead of sending both solved and critical positions, we simply choose ONLINE-CP for further analysis, i.e. all instances of online fine-tuning solver for the rest of this section refers to ONLINE-CP. In conclusion, these results indicate that  $\theta_0$  provides less accurate heuristics, which impacts the proof search negatively. By performing online fine-tuning with either solved or critical positions, we can fine-tune the PCN dynamically according to the manager's current focus and therefore find faster solutions.
Table 1: The number of nodes and time to solve 16 7x7 Killall-Go three-move openings by the baseline and three variants of online fine-tuning solvers. "# Nodes" lists the numbers of all nodes visited by the manager and workers together. All the listed times are rounded to the nearest second. The rightmost column lists the number of PCN models produced by the online fine-tuning trainer.
BASELINE	ONLINE-SP	ONLINE-CP	ONLINE-SP+CP
# Nodes	Time (s)	# Nodes	Time (s)	# PCN	# Nodes	Time (s)	# PCN	# Nodes	Time (s)	# PCN
JA	8,964,444,959	142,115	4,054,562,593	69,699	359	1,288,601,416	22,384	186	1,425,668,707	24,865	225
JB	7,137,514,712	155,786	3,378,672,517	83,454	424	1,576,437,139	31,957	272	1,601,479,130	31,455	283
JC	721,004,784	12,514	819,264,890	13,963	57	316,391,324	6,537	59	414,108,746	8,343	69
JD	1,271,426,148	30,209	846,365,092	19,396	113	545,655,175	11,083	102	502,966,563	10,896	103
KA	134,881,952	2,103	143,814,448	2,621	14	111,838,889	2,102	18	104,905,173	1,931	18
KB	10,153,035,632	156,583	3,794,290,131	64,493	305	2,242,789,149	38,947	343	2,527,488,112	43,200	386
KC	38,217,263	747	45,217,101	1,156	6	26,441,989	758	6	25,508,784	706	6
KD	2,754,213,379	47,494	1,504,977,329	25,715	126	955,257,191	17,434	145	920,902,808	16,357	148
KE	1,197,819,407	18,771	214,614,577	3,917	21	181,418,954	3,336	30	168,590,287	3,095	28
KF	9,516,440,320	147,271	6,080,836,868	100,690	519	2,107,185,330	35,418	285	2,027,558,505	35,197	305
DA	7,322,743,383	112,874	3,015,438,589	50,046	248	1,761,842,477	30,313	266	1,665,511,033	28,337	235
SA	51,272,288	937	54,574,495	1,471	7	41,863,480	992	9	41,796,555	1,105	10
SB	215,380,103	3,860	65,970,358	1,423	7	55,541,455	1,364	12	109,591,487	2,258	20
SC	97,559,402	1,557	213,889,777	3,553	19	98,661,355	1,715	16	93,535,813	1,655	15
SD	8,187,017,679	124,644	3,821,472,453	63,058	329	1,395,444,447	23,751	154	1,485,439,307	25,531	224
SE	4,297,808,879	64,227	2,065,528,927	33,437	166	757,256,934	12,465	103	1,200,741,176	20,428	182
sum	62,060,780,290	1,021,692	30,119,490,145	538,092	-	13,462,626,704	240,556	-	14,315,792,186	255,359	-
Table 1 also shows another interesting result: the larger the problem, the better the improvement. For better visualization, the solving times are depicted as a bar chart in Figure 4, where the x-axis is sorted according to the solving time of the baseline solver. In Figure 4, the online fine-tuning solver solves all openings within 40,000 seconds, while the baseline solver uses more than one day to solve six openings. Most impressively, for JA, the online fine-tuning solver performs about 6.35 times faster than the baseline, reducing the computation time from 142,115 to 22,384 seconds, while the number of visited nodes is reduced from 8.96 billion to 1.29 billion or so nodes.
The online fine-tuning solver does not always perform better than the baseline, especially when the openings are relatively easy to solve for the baseline. In Table 1, the baseline uses less time to solve SA and SC. The following reasons may be why this limitation exists for smaller problems. First, the
Figure 4: Solving time comparison for 16 three-move 7x7 Killall-Go openings.
(a) Sub-position of  $JA$
Figure 5: Behavioral analysis of the online fine-tuning solver for the opening JA.
(b) Average length of critical positions.
online fine-tuning solver relies on a trainer to fine-tune the PCN. The quicker the problem can be solved, the less time the trainer has to fine-tune specific PCN weights. Consequently, it has a weaker impact on improvement. This is corroborated by the fact that these two openings end up with less than 20 PCN versions,[5] as shown in Table 1. Second, when compared to the baseline, the online fine-tuning solver has less computing power for the workers. The trainer overhead takes up a GPU and leaves workers with seven instead of eight GPUs used by the baseline.
It is also worth mentioning that all three-move openings in both jump and stretch are solved. As these openings are considered the most difficult moves, we expect that both jump and stretch (i.e., the two-move openings) can probably be solved completely in the near future, with more computing resources.
4.3 Behavior analysis for the online fine-tuning solver
We further analyze the behavior of two solvers by studying the opening  $JA$ , where the online fine-tuning solver has the largest speedup among all openings, i.e. 6.35 times faster than the baseline. We observe several positions in which the winning moves for White differed between the two solvers. For example, a crucial sub-position in the solution tree is shown in Figure 5a. The baseline and online fine-tuning solver chose moves A and B to search, respectively. We examine  $\theta_0$  and find that the
In our settings, the trainer typically generates a new PCN version about every 120 seconds.
probabilities from the policy network for moves A and B are 0.416 and 0.133, respectively. As a result, the baseline solver has a lower chance to explore B. In contrast, with the help of the OFT, the online fine-tuning solver quickly realizes that solving B is faster, though it still attempted to search A initially, as it starts with the same  $\theta_0$ . In terms of node counts, the baseline spent a total of 1.63 billion nodes, with approximately 1.48 billion (91.00%) and 35.50 million (2.18%) nodes spent on searching A and B, respectively, while the online fine-tuning solver spent a total of 136.47 million nodes on that position, with approximately 22.13 million (16.21%) and 95.17 million (69.74%) nodes spent on searching A and B, respectively. This example clearly demonstrates the advantage of using the OFT.
Next, we investigate the set of critical positions maintained by the online fine-tuning trainer, as described in subsection 3.2.2. During fine-tuning, the trainer randomly selects positions from the queue, then runs self-play games from these positions. Figure 5b shows the average path length of critical positions during training in the y-axis, and the training iteration in the x-axis, where the trainer generates a new PCN version for every iteration. The length starts at around 10 and gradually increases to nearly 25 in the end. This is because as the manager search tree grows through node expansion, the critical positions are chosen from deeper parts of the tree. The curve also fluctuates as the proof search progresses. This is because the manager tends to focus on a sub-problem at a time. When a subtree is solved, the manager may then shift its attention to other unsolved parts of the proof search, which can have a relatively shallower depth. We also analyze similar figures for other openings in the appendix.
4.4 Updating PCNs in online fine-tuning
We investigate the impact of updating PCNs for the manager and workers during online fine-tuning. We select four openings, JC, KE, DA, and SE, one from each opening group, for this experiment. Table 2 summarizes results, where BASELINE denotes the baseline solver, ONLINE-CP denotes the online fine-tuning solver that updates PCNs for both the manager and workers (as described in subsection 3.2.2), ONLINE-CP-M denotes updating the PCN for the manager only, and ONLINE-CP-W for workers only.
First, updating PCNs for both the manager and workers performs the best. By using consistent PCNs, jobs assigned by the manager are efficiently solved by the workers. With inconsistent PCNs, the results can be even worse than the baseline. Generally, ONLINE-CP-M outperforms ONLINE-CP-W, except for the opening SE. We find that the pre-trained PCN causes ONLINE-CP-W to divide its computing resources across several OR nodes (white moves), many of which are relatively difficult to solve. In contrast, for ONLINE-CP-M, with the updated PCNs, the manager focuses on one white move (or a smaller number of white moves), of which they are much easier to solve. Thus, even if there is a mismatch between the manager and workers in ONLINE-CP-M, focusing on one good white move can still result in efficiency.
Table 2: Impact of updating PCNs in the online fine-tuning solver. The number represents the time (in seconds) for solving each opening by using different methods. Table 3: Ablation study for job assignment schemes in the online fine-tuning solver. The number represents the time (in seconds) for solving each opening by using different methods.
JC	KE	DA	SE
BASELINE	12,514	18,771	112,874	64,227
ONLINE-CP-M	9,992	5,445	46,719	28,769
ONLINE-CP-W	13,516	54,360	65,304	26,168
ONLINE-CP	6,537	3,336	30,313	12,465
w/o AND assg.
w/o top-k
w/o AND assg. & top-k
4.5 Ablation study for job assignment schemes
We conduct an ablation study in the online fine-tuning solver to analyze the impact of job assignment schemes, described in subsection 3.3. We only include ablations for top- $k$  selection and AND-player job assignment, since virtual solving is required to avoid job redundancy. The ablation study is performed on the same four openings as subsection 4.4, JC, KE, DA, and SE. Table 3 summarizes the ablation results, where ONLINE-CP denotes the online fine-tuning solver that uses both schemes; the other three versions denote the ablations by removing specific schemes from ONLINE-CP. If the top- $k$
selection is removed, the manager always selects the best child during selection for both AND-nodes and OR-nodes. If we do not follow the AND-player job assignment scheme (abbreviated as AND assign in the table), the manager assigns both AND-player and OR-player jobs.
From Table 3, ONLINE-CP performs the best in general. In particular, ONLINE-CP only requires around  $48.37\%$  of the computing time on average over all four openings, compared to the solver without both schemes (the last row in the table). When comparing each technique individually, the improvement varies from problem to problem.
This paper demonstrates the potential of using online fine-tuning for game solving. On average across multiple openings, our proposed online fine-tuning solver only uses  $23.54\%$  of the computation time compared to the baseline. Our distributed game solver is the first online fine-tuning method for problem solving based on AlphaZero-like algorithms. Although we focus on online fine-tuning throughout this paper, we can also claim that the complete distributed game solver is a life-long learning system. The online trainer continuously refines heuristics for unfamiliar (unsolved) positions, and forgets the previously learned knowledge (solved positions). However, this forgotten knowledge is saved (remembered) in the manager's solution tree. As a result, the worker and trainer will not need to evaluate these positions again.
There are many other topics for future investigation. Our experiments on the four challenging 7x7 Killall-Go opening groups show that two groups are likely to be solved in the near future. However, for the other two, or even 7x7 Killall-Go in its entirety, we expect more novel techniques are needed. As for the standard Go game, the largest solved board to date is only 5x6 in size [41], with no published progress in 14 years. We expect online fine-tuning to be one of the key improvements that can help push this boundary. As for generality, our method is not limited to Go but can be easily applied to other two-player zero-sum games like Hex or Othello. Moreover, we expect it has the potential to extend to single-player games such as Rubik's Cube, or even to other non-game fields, such as automated theorem proving [17] or chemical syntheses [42, 43].
This research is partially supported by the National Science and Technology Council (NSTC) of the Republic of China (Taiwan) under Grant Number NSTC 111-2222-E-001-001-MY2, NSTC 111-2221-E-A49-101-MY2, NSTC 110-2221-E-A49-067-MY3, and NSTC 111-2634-F-A49-013.
[1] H Jaap van den Herik, Jos W H M Uiterwijk, and Jack Van Rijswijck. Games solved: Now and in the future. Artificial Intelligence, 134(1):277-311, 2002.
[2] Jonathan Schaeffer. One Jump Ahead: Challenging Human Supremacy in Checkers. ICGA Journal, 20(2):93-93, 1997.
[3] Murray Campbell, A Joseph Hoane Jr, and Feng-hsiung Hsu. Deep Blue. Artificial intelligence, 134(1-2):57-83, 2002.
[4] David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of Go with deep neural networks and tree search. Nature, 529(7587):484-489, 2016.
[5] Oriol Vinyals, Igor Babuschkin, Wojciech M Czarnecki, Michael Mathieu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782):350-354, 2019.
[6] David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of Go without human knowledge. Nature, 550(7676):354-359, 2017.
[43] Akihiro Kishimoto, Beat Buesser, Bei Chen, and Adi Botea. Depth-First Proof-Number Search with Heuristic Edge Cost and Application to Chemical Synthesis Planning. In 33rd Advances in Neural Information Processing Systems, NIPS 2019, pages 7224–7234, 2019.
[44] Andrew J Palay. Searching with Probabilities. PhD thesis, Carnegie Mellon University, 1983.
[45] Akihiro Kishimoto and Martin Müller. A solution to the GHI problem for depth-first proof-number search. Information Sciences, 175(4):296-314, 2005.
A.1 PCN training
We basically follow the same PCN training method by Wu et al. [14] but replace the AlphaZero algorithm with the Gumbel AlphaZero algorithm [38], where the simulation count is set to  $32^6$  in self-play and starts by sampling 16 actions. The architecture of the PCN contains three residual blocks with 256 hidden channels. A total of 400,000 self-play games are generated for the whole training. During optimization, the learning rate is fixed at 0.02, and the batch size is set to 1,024. The PCN is optimized for 500 steps for every 2,000 self-play games. The pre-trained PCN requires around 13 hours to train on a machine with four 1080Ti GPUs, i.e. 52 1080Ti GPU-hours. For the online trainer, we use the same hyperparameters as the pre-trained PCN but only use one GPU.
A.2 7x7 Killall-Go solver
Our solver is built upon the state-of-the-art (SOTA) 7x7 Killall-Go solver [13] except for the following three changes. First, our solver uses PCN as heuristics while the SOTA solver trains a network with Faster to Life (FTL) techniques. Both networks aim to provide a faster move for solving, but FTL requires additional  $(komi^7)$  settings in solving, so PCN is much easier to use in our solver. Second, we implement the transposition table based on Shih et al. [39]. This greatly reduces the solving time. Finally, we implement a solution for resolving Graph-History-Interaction (GHI, i.e. cycles in Go) [44] problems to ensure the correctness of reusing solutions in the transposition table, based on Kishimoto and Müller [40, 45]'s GHI solution.
A.3 Worker design
The worker is itself a Killall-Go solver. It is GPU bound, i.e. it relies on GPUs more than CPUs since the PCN (a neural network) requires intensive GPU computation. Thus, to fully utilize GPU resources, we implement batch GPU inferencing to accelerate PCN evaluations for workers. In practice, we collect 48 workers together in one process with multiple threads. The process runs MCTS selection for each worker independently. Namely, a total of 48 leaf nodes are generated and evaluated by PCN with one GPU at once. The 48 leaf nodes are collected as a batch for batch GPU inferencing, with a batch size of 48. This method greatly reduces the solving time when more workers are used. The baseline distributed game solver creates eight processes as workers, each with one GPU, for a total of 384 workers (eight processes with 48 workers). The online fine-tuning solver has the same number of workers for fairness, but uses seven GPUs (one GPU is spared for the online trainer); the configuration is six processes with 55 workers and one process with 54 workers.
All experiments are conducted in three machines, each equipped with two Intel Xeon E5-2678 v3 CPUs, 192G RAM, and four GTX 1080Ti GPUs. We list other hyperparameters in Table 4.
For the memory used in solving, the manager requires 20G RAM for expanding every 1M nodes, and every 48 workers together in one process requires 30G RAM at most. Note that workers use the same amount of memory regardless of problem size. They are limited to 100,000 nodes per job; the job result is "unsolved" if a solution is not obtained within that limit.
Specifically, for BASELINE with 384 workers, solving  $KA$  used 2,103 seconds, required 3G RAM for the manager and 240G RAM for the workers; solving  $KB$  used 156,583 seconds, required 170G RAM for the manager and 240G RAM for the workers. However, for BASELINE with only 48 workers, solving  $KA$  used 12,151 seconds but only required 2G RAM for the manager and 30G RAM for the workers. Overall, the settings can be varied depending on available machines.
The original PCN training used 400 simulation counts in the self-play, requiring much more computing resources than using Gumbel algorithm.
<sup>7</sup>Since Black plays the first stone in the game of Go, White usually earns some extra points called komi for balance.
Table 4: Hyperparameters used in the baseline and online fine-tuning solvers. All variants of online fine-tuning solvers use the same settings.
BASELINE	ONLINE
Manager	# GPUs	1	1
vthr	16.5	16.5
k for top-k selection	4	4
Worker	# GPUs	8	7
# workers	384	384
# node limitation per job	100,000	100,000
Trainer	# GPUs	0	1
B.2 Scalability of the distributed game solver
To evaluate the scalability of the distributed game solver, we run BASELINE with different numbers of workers on  $KA$ . Specifically, the solvers use 384, 192, 96, and 48 workers, using 8, 4, 2, and 1 GPU, respectively. Every 48 workers share one GPU. The results are shown in Table 5. Overall, the speedup is around 1.8 times faster when the number of workers is doubled (up to 384 workers due to our machine limitation).
Table 5: Detailed statistics for solving  ${KA}$  by BASELINE with different numbers of workers.
# Workers	# Nodes	Time (s)	Manager # Nodes	# Jobs	Avg. Job Time (s)	Avg. Job # Nodes	#PCN	Solved Jobs (%)	Avg. Worker Loading (%)	Speedup
384	134,881,952	2,103	121,236	21,748	34.48	6,196.46	0	97.87%	94.53%	5.78
192	120,676,465	3,596	99,678	18,598	35.92	6,483.32	0	98.44%	98.57%	3.09
96	112,344,894	6,502	84,752	16,422	37.45	6,835.96	0	98.87%	98.90%	1.71
48	109,362,406	12,151	74,665	15,292	37.78	7,146.73	0	99.05%	98.60%	1.00
B.3 Statistics of solving 7x7 Killall-Go three-move openings
Figure 6 shows the next winning moves (the fourth moves) of 16 three-move openings for both baseline and ONLINE-CP solvers. Generally, both solvers solve the openings at the same next moves, except  $JB$ . The full solution trees for each opening can be found in this link: https://rlg.iis.sinica.edu.tw/papers/neurips2023-online-fine-tuning-solver/solution-trees. We also provide a tool and a README file for explaining the solution tree.
It is worth mentioning that  $JA$  and  $JB$  are similar to one of the common josekis $^{8}$  played in 19x19 Go. The joseki usually occurs when Black makes a corner enclosure move, also known as shimari in Japanese, like the two stones marked as “1” in  $JA$  and  $JB$ . Then, White attempts to invade Black's territories by playing at the stone marked as “2”. Judging by the online fine-tuning solver's ability to solve  $JA$  and  $JB$ , we foresee a high potential to extend our work to solving other 19x19 Go corner josekis in the future.
In addition, Figure 7 shows the curve for average critical position lengths. These curves are all similar in the sense that it starts with small average lengths, which gradually increases during fine-tuning.