Solving the Rubik's Cube with Approximate Policy Iteration

1 INTRODUCTION . The Rubik ’ s Cube is a classic combination game that poses unique and interesting challenges for AI and machine learning . Although the state space is astronomically large ( 4.2× 1019 different states for the 3x3x3 cube ) , only a single state is considered solved . Furthermore , unlike the game of Go or Chess , the Rubik ’ s Cube is a single-player game and a sequence of random moves , no matter how long , is unlikely to end in the solved state . Developing reinforcement learning algorithms to deal with this property of the Rubik ’ s Cube might provide insight into other sparse-reward environments . While methods for solving the Rubik ’ s Cube have been developed , only relatively recently have methods been derived that can compute the minimal number of moves required to solve the cube from any given starting configuration ( Rokicki et al. , 2014 ; Rokicki , 2014 ) . In addition , the Rubik ’ s Cube and its solutions are deeply rooted in group theory , raising interesting and broader questions about the applicability of machine learning methods to complex symbolic systems , including mathematics . Finally , the classical 3x3x3 Rubik ’ s Cube is only one representative of a much larger family of possible combination puzzles , broadly sharing the characteristics described above and including : ( 1 ) Rubik ’ s Cubes with longer edges ( e.g . 4x4x4 ) ; ( 2 ) Rubik ’ s Cubes in higher dimensions ( e.g . 2x2x2x2 ) ; as well as ( 3 ) Rubik ’ s cube on non-cubic geometries ( Pyriminix , etc ) and their combinations . As the length of the sides and dimensions are increased , the complexity of the underlying combinatorial problems rapidly increases and , for instance , God ’ s numbers for the 4x4x4 cube is not known . In short , for all these reasons , the Rubik ’ s Cube and its variations pose interesting challenges for machine learning . Here we develop deep reinforcement learning methods , in particular a new form of Approximate Policy Iteration ( API ) , for addressing these challenges . ∗Equal contribution Approximate Policy Iteration ( Bertsekas & Tsitsiklis , 1995 ; Bagnell et al. , 2004 ; Kakade & Langford , 2002 ; Scherrer , 2014 ; Lazaric et al. , 2010 ) is a core Reinforcement Learning ( RL ) algorithm . Recently , a new type of API algorithm , called Dual Policy Iteration ( Sun et al. , 2018 ) , has achieved success in two player zero-sum games such as Go , Chess , Shogi and Hex ( Anthony et al. , 2017 ; Silver et al. , 2017a ; c ) . AlphaZero ( Silver et al. , 2017c ) and ExIt ( Anthony et al. , 2017 ) are examples of Dual Policy Iteration . These algorithms update the policy in a more sophisticated way than traditional API methods by using two policies : a fast policy ( usually a neural network ) and a slow policy ( usually tree search ) . The fast policy is trained via supervised learning on data generated from gameplay from the slow policy . The slow policy then uses the fast policy to guide the tree search . In this way , the fast policy is used to improve the slow policy , and the slow policy generates better data to train the fast policy . This work is the first to solve the Rubik ’ s Cube with reinforcement learning . Although DPI works for two-player games , our work is the first DPI algorithm to succeed in an environment with a high number of states and a small number of reward states . A number of these `` needle in a haystack '' environments such as generating meaningful sentences or code are seen as long-term goals for the field of AI . There is no clear way to apply current DPI algorithms such as AlphaZero or ExIt to sparse-reward environments such as the Rubik ’ s Cube . This is because a randomly initialized policy will be unlikely to encounter the single reward state . This will cause the value function to be biased or divergent and the fast policy will not converge to the optimal policy . In our work we overcome this problem by training the fast policy on a distribution of states that propagate the reward signal to from the goal state to the further states . Our algorithm , called Autodidactic Iteration ( ADI ) , trains a neural network value and policy function through an iterative process . These neural networks are the `` fast policy '' of DPI described earlier . In each iteration , the inputs to the neural network are created by starting from the goal state and randomly taking actions . The targets seek to estimate the optimal value function by performing a breadth-first search from each input state and using the current network to estimate the value of each of the leaves in the tree . Updated value estimates for the root nodes are obtained by recursively backing up the values for each node using a max operator . The policy network is similarly trained by constructing targets from the move that maximizes the value . After the network is trained , it is combined with MCTS to effectively solve the Rubik ’ s Cube . We call the resulting solver DeepCube . 2 RELATED WORK . Erno Rubik created the Rubik ’ s Cube in 1974 . After a month of effort , he came up with the first algorithm to solve the cube . Since then , the Rubik ’ s Cube has gained worldwide popularity and many human-oriented algorithms for solving it have been discovered ( Ruwix ) . These algorithms are simple to memorize and teach humans how to solve the cube in a structured , step-by-step manner . Human-oriented algorithms to solve the Rubik ’ s Cube , while easy to memorize , find long suboptimal solutions . Since 1981 , there has been theoretical work on finding the upper bound for the number of moves necessary to solve any valid cube configuration ( Thistlethwaite , 1981 ; Reid , 1995 ; Radu , 2007 ; Kunkle & Cooperman , 2007 ; Rokicki , 2010 ) . Finally , in 2014 , it was shown that any valid cube can be optimally solved with at most 26 moves in the quarter-turn metric , or 20 moves in the half-turn metric ( Rokicki et al. , 2014 ; Rokicki , 2014 ) . The quarter-turn metric treats 180 degree rotations as two moves , whereas the half-turn metric treats 180 degree rotations as one move . For the remainder of this paper we will be using the quarter-turn metric . This upper bound on the number of moves required to solve the Rubik ’ s cube is colloquially known as God ’ s Number . Algorithms used by machines to solve the Rubik ’ s Cube rely on hand-engineered features and group theory to systematically find solutions . One popular solver for the Rubik ’ s Cube is the Kociemba two-stage solver ( Kociemba ) . This algorithm uses the Rubik ’ s Cube ’ s group properties to first maneuver the cube to a smaller sub-group , after which finding the solution is trivial . Heuristic based search algorithms have also been employed to find optimal solutions . In 1985 , it was shown that iterative deepening A * ( IDA * ) could be used to find optimal solutions to combination puzzles ( Korf , 1985 ) . This algorithm was then combined with pattern databases ( Culberson & Schaeffer , 1998 ) to successfully find optimal solutions to the Rubik ’ s Cube ( Korf , 1997 ) . Later , ( Sturtevant & Chen , 2016 ) used advances in compute power ( 5TB of external memory and parallel processing ) , to show that bidirectional brute-force search could find optimal solutions to the Rubik ’ s Cube in up to 28 hours . More recently , there has been an attempt to train a DNN – using supervised learning with hand-engineered features – to act as an alternative heuristic ( Brunetto & Trunda , 2017 ) . These search algorithms , however , take an extraordinarily long time to run and usually fail to find a solution to randomly scrambled cubes within reasonable time constraints . Besides hand-crafted algorithms , attempts have been made to solve the Rubik ’ s Cube through evolutionary algorithms ( Smith et al. , 2016 ; Lichodzijewski & Heywood , 2011 ) . However , these learned solvers can only reliably solve cubes that are up to 5 moves away from the solution . We solve the Rubik ’ s Cube using API ( Bertsekas & Tsitsiklis , 1995 ; Bagnell et al. , 2004 ; Kakade & Langford , 2002 ; Scherrer , 2014 ; Lazaric et al. , 2010 ) . API is a general purpose reinforcement learning algorithm , and includes API with stochastic policies ( Schulman et al. , 2015 ; Kakade , 2001 ; Bagnell & Schneider , 2003 ; Baxter & Bartlett , 2001 ) , conservative API ( Kakade & Langford , 2002 ) , and API with learned critics ( Rummery & Niranjan , 1994 ) . Even though many of these API algorithms have certain theoretical optimality guarantees , they rely on random exploration ( such as REINFORCE ( Williams , 1992 ) -type policy gradient or -greedy ) and as a result are not sample-efficient . Dual Policy Iteration ( Sun et al. , 2018 ) algorithms such as AlphaZero and Exit are a new type of API algorithm that has achieved success in two player zero-sum games such as Go , Chess , Shogi and Hex ( Anthony et al. , 2017 ; Silver et al. , 2017a ; c ) . These algorithms use two policies : a fast policy and a slow policy . The fast policy is trained via supervised learning on data generated from gameplay from the slow policy . The slow policy then uses the fast policy to guide the tree search . The main difference between DPI and traditional API algorithms is the use of a forward dynamics model . With the use of this model , the slow policy can perform a tree search , abandoning the need for random exploration . Using this forward dynamics model allows for improved sample efficiency compared to traditional API algorithms . It is now well-known that reinforcement learning with nonlinear function approximation is prone to diverge ( Baird , 1995 ; Boyan & Moore , 1995 ; Gordon , 1995 ) . Wang & Dietterich ( 1999 ) approaches the problem of learning a stable value function with nonlinear approximation for combinatorial optimization by using a composite error function . Learning a value function with nonlinear approximation within API is not much of a problem when trying to solve Go or Chess . This is because the problem is simply binary classification based on the current board state . With the Rubik ’ s Cube , on the other hand , approximating the value function is very difficult because even a naive monte-carlo approach would not work since it will never encounter the solved state . ADI ensures stable and approximately unbiased training of the value function by sampling states in a way that ensures the goal state ’ s learning signal will propagate during the training . This approach is somewhat similar to the method proposed in Boyan & Moore ( 1995 ) . 3 THE RUBIK ’ S CUBE . The Rubik ’ s Cube consists of 26 smaller cubes called cubelets . These are classified by their sticker count : center , edge , and corner cubelets have 1 , 2 , and 3 stickers attached respectively . There are 54 stickers in total with each sticker uniquely identifiable based on the type of cubelet the sticker is on and the other sticker ( s ) on the cubelet . Therefore , we may use a one-hot encoding for each of the 54 stickers to represent their location on the cube . However , because the position of one sticker on a cubelet determines the position of the remaining stickers on that cubelet , we may actually reduce the dimensionality of our representation by focusing on the position of only one sticker per cubelet . We ignore the redundant center cubelets and only store the 24 possible locations for the edge and corner cubelets . This results in a 20x24 state representation which is demonstrated in Figures 2c and 2d . Moves are represented using face notation originally developed by David Singmaster : a move is a letter stating which face to rotate . F , B , L , R , U , and D correspond to turning the front , back , left , right , up , and down faces , respectively . A clockwise rotation is represented with a single letter , whereas a letter followed by an apostrophe represents a counter-clockwise rotation . For example : R and R ’ would mean to rotate the right face 90◦ clockwise and counter-clockwise , respectively . Formally , the Rubik ’ s Cube environment consists of a set of 4.3× 1019 states S which contains one special state , ssolved , representing the goal state . At each timestep , t , the agent observes a state st ∈ S and takes an action at ∈ A with A : = { F , F ’ , . . . , D , D ’ } . After selecting an action , the agent observes a new state st+1 = A ( st , at ) and receives a scalar reward , R ( st+1 ) , which is 1 if st+1 is the goal state and −1 otherwise . It is important to note that this environment is deterministic with known forward and backwards dynamics .

Summary: The authors show how to solve the Rubik cube using reinforcement learning (RL) with Monte-Carlo tree search (MCTS). As common in recent applications like AlphaZero, the RL part learns a deep network for policy and a value function that reduce the breadth (policy) and depth (value function) of the tree searched in MCTS. This basic idea without extensions fails when trying to solve the Rubik cube because there is only one final success state so the early random policies and value functions never reach it. The solution proposed by the authors, called autodidactic iteration (ADI) is to start from the final state, construct a few previous states, and learn value function on this data where in a few moves a good state is reached. The distance to the final state is then increased and the value function learn more and more. This is an interesting idea that solves the Rubik cube, but the paper lacks a more detailed study. What other problems can be solved like this? Would a single successful trajectory be enough to use it in a wider context (as in https://blog.openai.com/learning-montezumas-revenge-from-a-single-demonstration/) ? Is the method to increase distance from final state specific to Rubik cube or general? Is the training stable with respect to this or is it critical to get it right? The lack of analysis and ablations makes the paper weaker.