Self-Supervision is All You Need for Solving Rubik's Cube

1 INTRODUCTION . Combinatorial search is a challenging task both in theory and in practice . Representative problems include Traveling Salesman Problem , in which a salesman tries to travel through multiple cities in the shortest path possible . Despite how easy it sounds , the problem is labeled NP-hard due to its combinatorial complexity ( Papadimitriou & Steiglitz , 1998 ) ; as the number of cities increases , the number of possible combinations skyrockets . In the real world , algorithms for such problems are applied in various ways to optimize logistics , supply chain , resource planning , etc . In the past , researchers have proposed several shortcut algorithms to tackle the combinatorial complexity of such problems . Recently , in particular , reinforcement learning has enabled automatic search of algorithms to solve combinatorial problems near-optimally ( Mazyavkina et al. , 2021 ) ; by rewarding efficient moves , DNNs can learn to find increasingly more optimal solutions . Rubik ’ s Cube is also one such problem ( Demaine et al. , 2018 ) , and there have been not only logical search methods ( Korf , 1997 ; Rokicki et al. , 2014 ) but also heuristic ( Korf , 1982 ; El-Sourani et al. , 2010 ; Arfaee et al. , 2011 ) and reinforcement learning approaches ( McAleer et al. , 2018 ; Agostinelli et al. , 2019 ; Corli et al. , 2021 ) that can solve any scrambled state of the puzzle . Nevertheless , combinatorial search is still no easy task ; you need detailed knowledge of the target problem , coding explicit logic , or configuring a reinforcement learning system . Reinforcement learning is especially notable for the need to adjust architecture , loss function , and hyperparameters several times to stabilize the learning process often involving non-differentiable operations . To overcome this difficulty , we developed a novel method to train DNNs in a self-supervised manner for solving combinatorial problems with a predefined goal . The method teaches a DNN probability distributions of optimal moves associated with problem states , with an assumption that , for a goal-predefined problem , random moves stemming from the goal state are probabilistically biased towards optimality . In our experiment , by training a DNN to solve Rubik ’ s Cube , we evaluate our method against an optimal solver and DeepCubeA ( Agostinelli et al. , 2019 ) . Contribution . We show that self-supervised learning can be all you need for solving some goalpredefined combinatorial problems near-optimally , without requiring specialized knowledge or reinforcement learning . 2 PROPOSED METHOD . In this section , we first provide an overview of the proposed method and then discuss its constraints . Below , we consider a combinatorial problem as a pathfinding task on an implicit graph . Each node in the graph represents a unique state and has edges as moves to its adjacent nodes with an equal probability . 2.1 OVERVIEW . The fundamental idea of the proposed method is as follows : 1 ) apply a sequence of random moves to the target problem , and 2 ) train a DNN to predict the last random move based on the problem state . By learning associations between problem states and probability distributions of the last moves , the DNN can infer step-by-step moves as a reverse path to the initial goal state . After training , the multinomial probability distribution of moves at inference step i should approximate P ( Xi = j : j ∈ M ) = ∑ k∈ M P ( Xi+1 = k ) · P ( Xi = j | Xi+1 = k ) ( 1 ) where Xi is a choice of move at step i , and j and k belong to the set of all available moves M at steps i and i+1 . Here , the choice at step i does not actually depend on the future step i+1 . Rather , it should approximate how similar states depended on past steps during training , which in turn can be viewed as future steps at inference . In this sense , the probability distribution of the move Xi implicitly depends on paths to the current state or similar that appeared during training . Figure 1 shows a miniature example . 2.2 CONDITION AND ASSUMPTIONS . There is one necessary condition for the proposed method to work , one assumption for a DNN to approximate towards an optimal solver , and another assumption for effective training and inference . First , to be solved with the proposed method , a problem must meet the following condition . Condition 1 . For the problem of interest , its goal state is predefined and readily available . For a DNN to learn probability distributions of moves connecting to the goal state , any path consisting of random moves must be derived from the goal state . Therefore , Condition 1 is necessary for the proposed method . Subsequently , the following two assumptions are desirable for a self-supervised DNN to approximate an optimal solver . Assumption 1 . The more optimal a move is , the more frequently it occurs . Although Assumption 1 is not necessary for applying the proposed method , the more true Assumption 1 is for the problem , the more optimal the DNN will be . Here , we formulate the statistical condition for Assumption 1 to be true on a problem when all moves at every node have an equal likelihood . Let pn , a be the probability of taking a particular move a as the first step to a specific target node in a path of n moves or more , pn , a = kmax∑ k=n Ck|n Ak ( 2 ) where k is the move count to the target state , kmax is the maximum number of k , A is the total number of possible moves at all nodes , and Ck|n is the number of k-move paths ( k ≥ n ) . Likewise , let b be an alternative to a that requires n + 1 moves or more , Assumption 1 is true if and only if pn , a ≥ pn+1 , b . Thus , provided pn , a = Cn|n/An + ∑kmax n+1 ( Ck|n/A k ) , the following must be satisfied for any n for Assumption 1 to be true : Cn|n An + kmax∑ k=n+1 Ck|n Ak ≥ kmax∑ k=n+1 Ck|n+1 Ak ( 3 ) At this point , both Ck|n and Ck|n+1 are unknown on an implicit graph , and the ∑ terms do not necessarily equal when the paths start with different moves . However , the only potential difference is the first move in two paths of different optimalities , and just taking a sub-optimal b move should not largely increase the total number of same-distance paths . Therefore , we assume that this holds true to varying degrees depending on the problem being addressed . Assumption 2 . The maximum number of moves required to solve any state—so-called God ’ s Number—or at least its upper bound should be known for a target problem . In the proposed method , let NG be God ’ s number of a given problem , the DNN receives problem states shuffled for 1 ∼ NG moves . If you estimate the number smaller than it actually is , then the DNN would not well adapt to problem states of higher complexities during inference . On the contrary , if you overestimate NG , you may unnecessarily apply additional moves to states of the highest complexity , resulting in more non-optimal moves in training scrambles . One way to address this limitation would be to keep increasing the maximum number of shuffles , as long as the DNN can make predictions beyond chance for the maximum complexity . If Condition 1 and Assumption 1 are met , random moves should have expected probability distributions peaking around optimal moves . It is also notable that the degree to which a problem satisfies Assumption 1 affects the optimality of the trained DNN : the more possible paths violating Assumption 1 , the less optimal the DNN will be . Although the example in Figure 1 has a very restricted setting for explanatory simplicity , this observation could apply to similar problems of higher dimensions and complexities under these constraints . Accordingly , we hypothesize DNNs to learn to solve goal-predefined combinatorial problems near-optimally . Last moveScramble 3 EXPERIMENT . 3.1 RUBIK ’ S CUBE . Rubik ’ s Cube is a cubic puzzle with 6 faces each having 9 color stickers . The goal of the puzzle is to move the stickers by rotating pieces on 6 faces , so that each face has stickers of only one color . Including the goal state , the puzzle can have approximately 4.3× 1019 different states . In our experiment , the puzzle is represented by its sticker locations and their colors . By assigning indices to all sticker locations and colors , one-hot encoding can represent the puzzle in 324 dimensions ( 54 sticker locations×6 colors ) . For rotating Rubik ’ s Cube , we employ the quarter-turn metric ( a 90° turn of a face counts as one move whereas a 180° turn counts as two ) , meaning 12 possible moves for every given state . In this configuration , solving Rubik ’ s Cube can be described as an implicit graph search problem in the 12-D action space . 3.2 LEARNING OBJECTIVE . Given the state of a Rubik ’ s Cube scrambled for random 1 ∼ 26 moves1 , a DNN predicts the last move of a scramble ( see Figure 2 ) . Here , we set the maximum scramble length to 26 because the number is known as sufficient for optimally solving any state of Rubik ’ s Cube ( Kunkle & Cooperman , 2007 ) . Since the inverse of the last scramble move is expectedly close to optimal under Assumption 1 , we hypothesize that the DNN can learn to solve the puzzle near-optimally through this learning objective . 3.3 MODEL . We employ the same architecture as DeepCube and DeepCubeA ( McAleer et al. , 2018 ; Agostinelli et al. , 2019 ) . The model first has two fully-connected ( FC ) layers , followed by four residual blocks ( He et al. , 2016 ) each containing two FC layers . Finally , for the prediction output , the model has a 12-D FC layer with softmax activation . The DNN is trained for 1 , 000 , 000 steps with a fixed batch size of 5 , 200 ( 200 states per scramble length × 26 scramble moves ) . We use Adam optimizer ( Kingma & Ba , 2014 ) with the initial learning rate of 10−4 for updating model weights . 1We exclude obviously redundant moves like R ( 90° clockwise turn on the right face ) following R ’ ( counterclockwise ) . 3.4 INFERENCE . For solving Rubik ’ s Cube with the trained model , we adopt a simple beam search . Starting from depth 1 with a scrambled state , at every depth i , the DNN predicts the next moves for every candidate state . Let k be the width of the beam search , we pass at most k candidate paths and corresponding states to the next depth i+ 1 , sorted by their expected probabilities . Also , for every potential move , if a move a is predicted to be worse than chance ( i.e. , p ( Xi = a ) < 1/12 ) , we exclude the path from the set of candidates before sorting . The search continues until any of the candidate states is solved , at which point the search depth matches the solution length .

Summary: The paper proposes learning to solve the Rubik's cube by randomly scrambling the cube and using a deep neural network to predict the random actions. The authors argue that, if optimal actions are more likely than random ones, that this method will be biased towards finding optimal solutions. When searching for a solution, the network predicts an action and the opposite action is taken, as the network predicts actions taken when scrambling from the solved state and now wants to return to the solved state.  When solving a cube, the authors use beam search and keep states based on the move probability. They experiment with keeping 10, 100, and 1000 states at each depth in the beam search. The authors use a dataset of 1000 cubes scrambled 100 times to test their method. The results show that, as the number of states keeping more states during beam search leads to shorter paths with the shortest average path being 20.5.