SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ti-Rong Wu, $^{1*}$  Hung Guei, $^{1*}$  Ting Han Wei, $^{2}$  Chung-Chin Shih, $^{1,3}$  Jui-Te Chin, $^{3}$  I-Chen Wu $^{3,4}$
$^{1}$ Institute of Information Science, Academia Sinica, Taiwan
$^{2}$ Department of Computing Science, University of Alberta, Canada
$^{3}$ Department of Computer Science, National Yang Ming Chiao Tung University, Taiwan
<sup>4</sup>Research Center for Information Technology Innovation, Academia Sinica, Taiwan
tirongwu@iis.sinica.edu.tw, hguei@iis.sinica.edu.tw, tinghan@ualberta.ca, rockmanray.cs02@nycu.edu.tw, pikachin.cs10@nycu.edu.tw, icwu@cs.nctu.edu.tw
Game solving is a similar, yet more difficult task than mastering a game. Solving a game typically means to find the game-theoretic value (outcome given optimal play), and optionally a full strategy to follow in order to achieve that outcome. The AlphaZero algorithm has demonstrated super-human level play, and its powerful policy and value predictions have also served as heuristics in game solving. However, to solve a game and obtain a full strategy, a winning response must be found for all possible moves by the losing player. This includes very poor lines of play from the losing side, for which the AlphaZero self-play process will not encounter. AlphaZero-based heuristics can be highly inaccurate when evaluating these out-of-distribution positions, which occur throughout the entire search. To address this issue, this paper investigates applying online fine-tuning while searching and proposes two methods to learn tailor-designed heuristics for game solving. Our experiments show that using online fine-tuning can solve a series of challenging 7x7 Killall-Go problems, using only  $23.54\%$  of computation time compared to the baseline without online fine-tuning. Results suggest that the savings scale with problem size. Our method can further be extended to any tree search algorithm for problem solving. Our code is available at https://rlg.iis.sinica.edu.tw/papers/neurips2023-online-fine-tuning-solver.
Playing and solving strategic games have served as drivers and major milestones [1] in artificial intelligence research. To master such games, the objective is often designed to optimize on the objective of maximizing the probability of winning. In the past several decades, researchers made significant progress in game playing, reaching super-human playing levels in many domains. Successful examples include Chinook (checkers) [2], Deep Blue (chess) [3], AlphaGo (Go) [4], and AlphaStar (StarCraft II) [5]. Furthermore, AlphaZero [6, 7] and MuZero [8] even boast generality by mastering a variety of games without requiring expert human knowledge. Although these learning-based agents have progressed dramatically in playing strength, there are no guarantees that their decisions are always correct [9, 10] in terms of game-theoretic value, which is defined as the outcome of the game given optimal play for both players. Game solving is this pursuit of finding game-theoretic values.
Game solving is a more difficult challenge than game playing. Many seemingly simple games have astronomically large state spaces, with no simple way of exploring this space. Here is where
advancements in game playing can aid game solving. Strong agents are commonly leveraged to evaluate positions, providing guidance and reducing the search space significantly. For example, the checkers program Chinook claimed to have reached super-human levels as early as 1996 [2], then about 10 years later, played an instrumental role in the proof that checkers is a drawn game [11]. Similarly, contemporary learning-based approaches such as AlphaZero are widely used to help reduce the search space for game solving. Game solvers that utilized AlphaZero include Hex [12], Go [13], Killall-Go [14], and the Rubik's cube [15, 16]. Such approaches are not limited to applications in games but extend to other non-game fields, like automated theorem proving [17].
However, a major issue still exists when using learning-based approaches to aid game solving. In the two-player, zero-sum setting, a simple description for a proof involves verifying that there is a winning move for the winner, for all possibilities played by the losing side; i.e. no matter how the loser plays, the winner must be able to respond correctly. However, most learning-based agents are trained along a strong line of play by both players, with some exploration to nearby states for robustness. Using AlphaZero as an example, training samples are generated via self-play by the best version of itself up to that point. Learning-based methods are powerful in that they generalize for previously unseen positions, but accuracy tends to drop the further you stray from training samples. To verify all possibilities on the losing side, the vast majority of positions we must evaluate during the search for a proof are therefore out-of-distribution. To illustrate, AlphaZero-like networks have been shown to make inconsistent or completely incorrect evaluations, simply by adding two meaningless stones to a position [10]. In another example, in the attempt to solve one of the hardest Go life-and-death (L&D) problems from the famous book Igo Hatsuyoron,[2] all AlphaZero-like programs failed. It was hypothesized that this was because these highly specific problems are rarely encountered during training [18].
This paper proposes applying online fine-tuning methods to train learning-based systems while solving games. In our proposed methods, during game solving, an online trainer is added so that the learned heuristic is improved as new data that is relevant to the solving task is generated. This is done by utilizing new information such as solved and critical positions in the current solver search tree. The trainer therefore can learn better heuristics dynamically, that are particularly fine-tuned for the upcoming evaluations. Experiments are conducted on 16 challenging 7x7 Killall-Go three-move openings, shown in Figure 3. We develop a distributed game solver with online fine-tuning, that is built upon the state-of-the-art 7x7 Killall-Go solver [13]. Experiment results show that the online fine-tuning solver can greatly reduce the search space by a factor of 4.61 on average. Namely, it searches only  $21.69\%$  of nodes, using  $23.54\%$  of the computation time, as compared to the offline solver. Most importantly, for larger problems, the online fine-tuning solver performs significantly faster than that without, which implies that our method scales with problem size.
2.1 Game solvers
A two-player zero-sum game is considered solved if we know of a winning strategy for either player which guarantees a winning outcome, regardless of how the opponent plays; i.e. the player must have at least one action that leads to a win, for all actions by the opponent. A winning strategy is often represented as an AND-OR tree called a solution tree [19], where the game positions with the winner to move are represented by OR-nodes, and those for the opponent by AND-nodes. Leaf nodes in a solution tree are all terminal positions where the outcome is a win.
A solver is a program that does a proof search and can identify a winning strategy or a solution tree, if found. Solvers often rely on heuristic tree search for games with large and complex game state spaces. Search algorithms such as alpha-beta search [20], proof number search (PNS) [21], or Monte Carlo tree search (MCTS) [22, 23] have all been shown to be successful. In addition, previous research has shown that none of the algorithms dominates the others [24, 25, 14].
Igo Hatsuyoron is a classic collection of L&D problems in Go, which demand complex calculations to solve. L&D problems are puzzles that test your ability to identify the safety of specific pieces in a given game position.
<sup>3</sup>We only consider "weak solutions" [1] in this paper, where different opening positions are treated as independent sub-games. Draws are also not considered, but can be determined via two searches, one for each player. If both outcomes are losses, then it must be a draw.
2.2 Distributed game solver
In cases where search spaces are too large for a single instance solver under reasonable time and memory constraints, multiple solvers are often run in parallel, forming a distributed computing system, to scale up the solving process. Examples of games solved by distributed computing include checkers [11], heads-up limit hold'em poker [26], breakthrough [27], Hex [28], and Connect6 [25, 29].
These distributed game solving/analysis systems, also known as distributed game solvers, have been presented commonly with two components, a manager and a set of workers. A manager divides the overall problem into smaller sub-problems, keeping only the initial portion of the search tree – the beginning of the game – in memory. As this search tree is expanded, the manager may decide to offload analysis of specific positions to its workers. These offloaded sub-problems are also called jobs [25, 29]. A worker computes jobs by taking as input, a specific position and any relevant parameters (e.g. time limits), then outputs either a solved or heuristic value for that position. A worker can be a single solver, a game engine, or even a combination of both.
For example, in the checkers proof, Chinook and another depth-first proof-number search [30] solver were combined as a heuristic. From the perspective of the manager, a solved job result, such as a proven win, loss, or draw, can be thought of as a terminal node in its solution tree. Unsolved jobs also provide useful information, such as heuristic values to determine likely win or likely loss, to guide further tree expansions, with the worker acting as a relatively expensive and accurate heuristic. Similar to the checkers proof, Connect6 openings have also been solved by encapsulating the task of solving and playing a position into a single job, which was then dispatched by a manager to a set of workers [25, 29, 31]. In all examples listed above, a centralized scheme is used where one manager coordinates between dozens to hundreds of workers.
2.3 Proof Cost Network
When using neural networks as heuristics in solving, recent research points out that there is room for improvement when using the value network learned from the AlphaZero algorithm[16, 14]. In a search tree, when several actions can reach a winning outcome, AlphaZero-trained networks have no preference for choosing one that wins fastest. This can increase the amount of computation significantly.
To address this challenge, the Proof Cost Network (PCN) [14] predicts a proof cost value, rather than a win rate. The cost value represents a logarithmically-normalized estimate of the number of nodes that are required to solve the position. Specifically, PCN adopts the AlphaZero training process and generates self-play games using the cost value to guide the MCTS toward faster winning moves. These self-play games are then used to update PCN's cost values. The resulting network will focus the proof search on actions with minimal cost. Experiments show that the proof cost value is highly correlated to problem difficulty, and can significantly improve solving capability.
3 Game solver with online fine-tuning
This section describes our methods for applying online fine-tuning to game solving. We chose an MCTS-based solver due to its popularity when integrating AlphaZero networks as heuristics [16, 13, 14]. However, it is worth noting that the methods presented in this paper are search-independent and can be readily applied to other search algorithms such as alpha-beta search or PNS.
3.1 Distributed game solver
Our distributed game solver consists of a PCN serving as its heuristic, a manager, and a set of workers. The manager maintains an MCTS rooted at the position to be solved. During the proof search, the manager follows PUCT [32] selection to traverse from the root to a leaf node. Next, the PCN estimates the cost of the selected leaf node, denoted by  $v_{l}$ .  $v_{l}$  is a heuristic value representing the log estimated number of positions that must be examined to solve this node. If the value is larger than a designated threshold, i.e.  $v_{l} \geq v_{thr}$ , its proof cost is considered too high to warrant a job. The manager will then continue to follow MCTS, expanding the node and backpropagating  $v_{l}$  to the root. Alternatively, if  $v_{l} < v_{thr}$ , the leaf node is highly likely to be solved outright by a worker, at which
Figure 1: The online fine-tuning game solver architecture.
Figure 2: A manager AND-OR tree with six MCTS selection paths. Positions  $d$  and  $j$  are solved in this tree.
point a job is created. Job granularity is therefore controlled with  $v_{thr}$ . Larger  $v_{thr}$  generates more difficult jobs with higher failure rates, while smaller  $v_{thr}$  leads to easier but more numerous jobs. A balanced  $v_{thr}$  should be set according to the game instance and worker capabilities.
Workers are game solving programs that are limited by specific constraints, say, a given time limit. To keep the heuristic consistent during the proof search, workers use the same PCN weights as the manager. If the job is solved within the given constraint, the worker returns the result, either a win or a loss, back to the manager; otherwise, it returns an unknown. Once the manager receives the job result, it updates the search tree accordingly. For unsolved jobs, the manager expands the nodes that generated the corresponding jobs. The interaction between the manager and the workers is shown between Figure 1(a) and 1(b).
3.2 Online Fine-Tuning Trainer
The Online Fine-Tuning Trainer (OFT) maintains the PCN during the proof search so that the manager and workers have access to ever improving heuristics. Without online fine-tuning, both the manager and workers simply use a fixed PCN, denoted by  $\theta_0$ , trained via the AlphaZero self-play process. The OFT starts with  $\theta_0$ , then fine-tunes the weights via updates during the proof search. To do this, the manager picks out solved and/or critical positions in its search tree, adds them to the list of training samples, then the OFT uses them to perform self-play and training. The fine-tuned PCN  $(\theta_1, \theta_2, \dots, \theta_t, \dots)$  can then be used to further guide the manager and workers towards a faster proof. The manager and workers update to the most recent  $\theta_t$  immediately when a new PCN checkpoint is trained by the OFT. The above iterative process is shown in Figure 1. Details are provided in the following sections.
3.2.1 Online fine-tuning trainer with solved positions
During the proof search, many previously unsolved positions may become solved in the manager's search tree. This new information can be used by the OFT to improve the accuracy of the PCN. Figure 2 provides an example of a manager's AND-OR search tree and six recent selection paths. In this example, positions  $j$  and  $d$  are marked as solved and sent to the OFT after the first and fourth selection, respectively. The OFT maintains a queue that stores these solved positions, as shown in Figure 1(c). Self-play games are generated as in normal PCN training [14]. However, in the optimization phase, the OFT randomly samples training data not only from the generated self-play games, but also from the queue of solved positions. For these solved positions, the cost values are always set to zero (i.e. solved to be a win, from the perspective of the OR-player), since no nodes need to be examined to solve the position. The OFT only samples  $10\%$  of training data from the solved queue during optimization to avoid overfitting, where the remaining  $90\%$  are sampled from self-play games. In addition, the queue only stores the most recent 1,000 solved positions received from the manager. During self-play, when using  $\theta_{t}$  to evaluate positions that are solved by the manager, it is highly likely to predict costs close to zero. From the AND-player's perspective, it favors moves that lead to larger costs to delay the OR-player's victory. Therefore, self-play naturally explores positions which have not yet been solved in the manager's search tree.
3.2.2 Online fine-tuning trainer with critical positions
Other than solved positions, we can also improve the PCN with specific positions of interest chosen from the manager's current search tree. Positions are considered critical if they are selected in the most recent MCTS iterations in the manager. For example, in the first selection in Figure 2, all positions  $r$ ,  $a$ ,  $g$ , and  $j$  in the MCTS selection path are considered critical positions. During self-play, the trainer randomly chooses one critical position, and performs self-play starting from that position. Thus,  $\theta_t$  can provide more accurate predictions for positions that proof search is currently exploring.
A more selective process can be used to improve the quality of critical positions. First, we can omit  $r$ , since self-play from  $r$  is already performed to train  $\theta_0$ . Ideally, we would prefer to focus on deeper unsolved positions. To achieve this, we only consider the leaf position in the selection path as critical. Also, the OFT maintains a queue in which only the recent 1,000 critical positions are stored. This way, the OFT can focus on the most urgent positions which are likely to be solved soon. As these positions are also usually sent to the workers (if the PCN value  $v \leq v_{thr}$ ), the workers can also take advantage of  $\theta_t$ . Next, we can omit leaf positions solved solely by the manager; i.e. leaf nodes that were solved not as jobs. For example,  $j$  will not be considered critical in the first selection in Figure 2. Since  $j$  is already solved, it is not necessary to perform self-play from that position. Only  $m$ ,  $n$ ,  $h$ , and  $i$  will be sent to the OFT as critical positions in the second, third, fifth, and sixth selection, respectively. Note that  $r$ ,  $a$ , and  $g$  were critical positions before  $b$  and  $k$  became critical, since the parent nodes are always expanded before their children. Thus, the trainer will gradually fine-tune the PCN by focusing only on deeper critical positions to help avoid redundancy during fine-tuning.
In summary, the pre-trained  $\theta_0$  learns general heuristics by exploring from empty games, while the online  $\theta_t$  refines its heuristics for specific positions of interest. As a side note, the fine-tuning process is related to the catastrophic forgetting phenomenon [33], as the focus is shifted from one part of the proof search to another. Interestingly, forgetting is not only acceptable in this context, but probably even preferred, because the heuristic only needs to be accurate for the part of the search space the manager is currently working on. Additionally, the two proposed methods are independent and can be combined. We evaluate these methods in our experiments.
3.3 Manager job assignment improvements
Job assignment refers to the manager's responsibility of dividing the overall problem into distinct jobs. Better job assignment schemes can eliminate redundancy and improve parallelism. We propose three techniques to further improve the efficiency of job assignment, which we call virtual solving, top-k selection, and AND-player job assignment.
Virtual solving. When a job is assigned to workers, we assume that the job result will be solved, even before it is actually returned by a worker. The virtually solved outcome is backpropagated as a normal job outcome. This technique has similar concepts to the virtual loss [34], virtual win [29], and Young Brothers Wait Concept (YBWC) [35], which were used to avoid repeatedly searching superfluous nodes during the proof search. For example, in Figure 2, assume the manager selects a path from  $r$  to a leaf  $h$  and assigns the job to a worker, at which point  $h$  is immediately marked as virtually solved. Its parent node  $b$ , an OR-node, is then also marked as virtually solved. Furthermore, if nodes  $a$ ,  $c$ , and  $e$  ( $d$  is solved already) are all solved or virtually solved, their parent  $r$  will also be marked as virtually solved. When the job result returns, the manager reverts the virtually solved markers and updates the status of all nodes accordingly. The virtual solving technique can provide a highly efficient job assignment scheme, in that the manager search tree can be nearly the same as the solution tree if most virtually solved nodes are indeed winning.
Top- $k$  selection. We exploit the fact that all child nodes must be solved for every AND-node to improve parallelism. At each AND-node, we select uniformly at random among the top  $k$  unsolved children that are likely to be sent off as jobs eventually, i.e. those with the top  $k$  highest PUCT scores. For example, in Figure 2, assume  $k = 2$  and nodes  $a$  and  $b$  are the top two children of AND-node  $r$ ; nodes  $k$  and  $l$  are the top two for  $g$ . Note that we omit node  $j$  because it is already solved. At node  $r$ , the manager selects between  $a$  and  $b$  with equal probability. Note that selections at OR-nodes remain unchanged. In addition, we only apply top- $k$  selection when the simulation count of the AND-node is larger than  $k$ . Top- $k$  selection improves parallelism by allowing the manager to assign more jobs simultaneously, when it is combined with virtual solving. We use  $k = 4$  in our experiments.
AND-player job assignment. We only distribute AND-nodes as jobs, i.e. OR-nodes are never assigned and are directly expanded in the manager. For example, in Figure 2, the OR-node  $b$  is not assigned as a job even if  $v_{b} < v_{thr}$ . The manager creates the AND-node  $h$  from the leaf node  $b$ , then assigns it to a worker as a job. The underlying intuition is that assuming the PCN policy head output is accurate as a move ordering heuristic, the first guess will often be the move that leads to a solution for OR-nodes. Therefore, by skipping OR-nodes job assignment entirely, the manager gains a 1-ply look ahead. In practice, all three job assignment schemes are applied simultaneously.
We demonstrate our online fine-tuning game solver by solving several three-move 7x7 Killall-Go openings. 7x7 Killall-Go is a variant of Go, where the rules are the same except that: (a) Black places two stones initially, and (b) Black wins if all white stones are killed; otherwise, White wins. Since White aims to live, winning specific openings for this variant is equivalent to solving a L&D problem. Many Go experts believe that 7x7 Killall-Go is a win for White. So far, no proof has been published yet. In this paper, we only focus on weakly solved games [36] in which White wins. Thus, White is considered the OR-player throughout.
4.1 The 7x7 Killall-Go solver
We build our 7x7 Killall-Go solver upon an AlphaZero training framework [37]. First, we pre-train a PCN  $\theta_0$  [14] to serve as heuristics for the game solver (starting from an empty board). We incorporate the Gumbel AlphaZero algorithm [38] into PCN training, since it performs equivalently well even with a small simulation count. This reduces the computation cost for online fine-tuning without compromising accuracy. The pre-training took around 52 1080Ti GPU-hours. Next, we incorporate several useful techniques into the solver to accelerate solving. This includes relevance zone-based search (RZS) [13], zone pattern tables [39], and GHI handling to deal with cycles in Go [40]. This solver is then used as workers in a distributed game solver. The manager is also based on the above solver, with the job assignment techniques added, as described in subsection 3.3. The OFT is similar to the PCN pre-training, but with fine-tuning as described in subsection 3.2.