Package: catchem-alpha-zero
FeaturesInstallationpip install catchem-alpha-zeroProject OverviewCatchemAlphaZero(CAZ) showcases techniques in artificial intelligence for solving two-player games without any human knowledge or input except for the rules of the game.  Any two-player deterministic game can, in principle, leverage this framework.  Little is needed beyond the ability to:get the games current legal movesplay a movedetermine if the game has terminated.The project includes:Alpha-Beta Minimaxwhich is well suited to games with a small to medium sized decision tree (or games for which one can easily evaluate a position's likelihood of winning)Monte Carlo Tree Searchwhich uses random simulation to estimate the value of a state,  and biases the search towards paths that seem most promisingAlphaZero: an attempt to replicate the brilliant work ofDeepMind, who used deep convolutional neural networks in conjunction with Monte Carlo Tree SearchTo significantly improve upon the MCTS algorithm.  The neural network learns the optimal policy entirely through self-play reinforcement learning.Supported GamesChessConnectX (Connect 4for boards of arbitrary sizes)Tic Tac ToePlay Chess AgainstCatchemAlphaZeroThe crowning glory of this project is chess.CatchemAlphaZerobundles a GUI application with the wheel,alpha, installed into your virtual environments Scripts folder. To launch the GUI, either click the application within the scripts folder or typealphainto the command line (having activated your virtual environment). The application will launch with a splashscreen presenting the following options:Play as WhitePlay as BlackPlay as BothSpectate Mode (watch CAZ play against itself)Key featuresMCTS Num Sims:Increase the size of the Monte Carlo Tree Search to increase the difficulty (1,000 sims corresponds to a chess ELO of about 1,800).Show Eval Barto see how CAZ evaluates the current positionAsynchronous Ponder:CAZ is fully asynchronous and will ponder on your thinking time. CAZ efficiently recycles the tree search between moves and explores whilst its opponent is thinking. Take time to think but remember that CAZ is thinking too!Watch:Clickhereto watch a demo of the application.How strong is CAZ?The currentEloof CAZ is not precisely known (and obviously varies with increasing number of MCTS simulations). The ultimate plan is to integrate CAZ as a Lichess bot. For now, the strength of CAZ has been calculated by playing friends and colleagues who (are far better at chess than me and) have generously offered to play the bot. To date, CAZ has not lost on 1,000 simulations andChess.comconsistently estimates its Elo to be in the range 1,900 to 2,300. My suspicion is that it is at the lower end of this spectrum - like AlphaZero, CAZ is strong at strategic play and closed positions but struggles with tactical play and endgames (where technique matters more).At the time of writing, the 'state of the art' is to use an Efficiently Updating Sparse Flat Neighborhood Network (SF NNUE). This is a smaller, faster CPU bound network that allows for deeper searches and therefore shines in tactical play as well as strategic play. This project uses a large convolutional ResNet architecture exactly as per DeepMind's specification. It works best on a GPU (but CPU only runs are supported as well).A friend (white) vs CAZ (Black) 0-1 under classical conditions. CAZ wins via resignation. Chess.comestimatedthe Elo scores as 1,800 (white), 2,350 (black).It's possible to train CAZ directly but if you'd like to use the weights that I have produced, you can download them fromhere. CAZ assumes that the weights are saved in aweightsfolder within the active current directory.How it worksCatchemAlphaZero is a project that explores artificial intelligence techniques for two player games. The project started withminimax, which was then extended toalpha-betaminimax.MCTSwas added to support games where leaf nodes could not be reached via brute force search. AlphaZero extends MCTS by using neural networks to both guide the search and provide and evaluation of each position rather than entering the rollout phase.CAZ produces beautiful visualisations for each game. In particular, it is possible to render the full tree search as well as the policy & evaluation associated with each state. The image below shows the output of a tree search with 10 simulations (please note that CAZ assumes that you have theGraphVizapplication already installed.)Rendered with ❤️ using CAZ. See the tutorial for details on how.CAZ also renders each game state as HTML and will show the policy associated with each state as an HTML grid with a heatmap overlay. The darker squares correspond to squares favoured by the policy. The evaluation shows that CAZ believes it has a 100% chance of winning from here and correctly identifies the three moves that can lead to a win.Heatmap policy for Tic-Tac-Toe. States and policied states are understood natively by IPython.Learning through self-play reinforcement learningCAZ is able to learn games entirely through self-play reinforcement learning. The idea is to use MCTS to explore a sample of the search space that seems most promising. At each step, the MCTS is guided by a policy retrieved from the neural network and evaluates a position using the neural network's evalution result. The neural network is a two-headed (policy head and value head)ResNetarchitecture that is described in detail in DeepMind'sNature paper. Below, is an illustration of the architecture used for TicTacToe. For chess, the architecture uses 19 residual network blocks:Visualisation of the network architecture for TicTacToe using Tensorboard.To improve the neural network, CAZ learns through self-play. The steps are:Play a game against itself relying on MCTS and the neural networkAt each step, improve the current policy via MCTS - MCTS is a policy improvement operatorRecord the outcome of the match after the game is played to completionPlay 100s of matches and batch the results togetherTrain the neural network against the match results. In particular, learn an improved policy based on the MCTS. Additionally, improve the network's evaluation function by assigning a score {-1, 0, +1} toeach statedepending upon the final outcome.If you'd like to train the network yourself, check out the command line interface below.Command line Interface FeaturesCAZ exposes five entry points via the command line:run,learn,kaggle,super,hyperRunCAZ from the terminal to see it play a gameLearnthrough self-play if you wish to train a netowrkKaggleenables you to run a Kaggle compliant agent that complies with the KaggleSimulation CompetitionsAPIsupertrains a neural network in chess through supervised learning. Whilst it is entirely possible to learn solely through self-play reinforcement learning, it would take a very long time to do this exclusively for chess (Deepmind used 5,000 TPUs for training which is out of my budget!). Supervised learning was used to accelerate the learning process specifically for chess but not for any other game.Hyperperforms a search over CAZ's various hyperparameters to ensure that it is learning optimally as training progressesExamples of how to run the various entry points are shown below:caz run --game=tic --player1=mcts --player2=az # MCTS vs AlphaZero algorithm.caz run --game=connectx --player1=mcts --player2=mctscaz run --game=chess --player1=az --player2=azcaz learn --game=connectxcaz kaggle --game=connectx --player1=mcts --player2=mctscaz supercaz hyper --game=connectx --gen=53Chess RepresentationCAZ follows Deepmind's proposal for representing input features and the action space per thefollow up paper. In particular, a state is represented by an 8x8x20 input tensor where the 20 planes correspond to:Current player's pawn positionsCurrent player's rook positionsCurrent player's knight positionsCurrent player's bishop positionsCurrent player's queen positionsOpposing player's king positionsOpposing player's pawn positionsOpposing player's rook positionsOpposing player's knight positionsOpposing player's bishop positionsOpposing player's queen positionsOpposing player's king positions8x8 matrix of 0's or 1's denoting the player's turn (all zeroes or all ones)8x8 matrix of 0's or 1's denoting current player's kingside castling rights8x8 matrix of 0's or 1's denoting current player's queenside castling rights8x8 matrix of 0's or 1's denoting opposing player's kingside castling rights8x8 matrix of 0's or 1's denoting opposing player's queenside castling rights8x8 matrix of showing the halfmove clockindicators for the position of any en passant squares8x8 matrix denoting whether the state is a two-fold repetitionThe action space is an 8x8x73 tensor where each 8x8 plane corresponds to:x7 N moves (the maximum number of squares you can move north is 7)x7 NE moves (again, moving 1-7 moves in a north-east direction)x7 E movesx7 SE movesx7 S movesx7 SW movesx7 W movesx7 NW movesKnight move NNE (i.e. one square east and two north)Knight move ENE (i.e. two square east and two north)Knight move ESEKnight move SSEKnight move SSWKnight move WSWKnight move WNWKnight move NNWPromote NW Knight (underpromote to a knight by capturing north-west)Promote N Knight (underpromote to a knight by advancing north)Promote NE KnightPromote NW RookPromote N  RookPromote NE RookPromote NW BishopPromote N  BishopPromote NE BishopVisualisation of the observation space (left) and action space (right). Here, CAZ is set to show the top two moves which align with common theory for the Fried Liver Attack. You can tweak the slider to show more/fewer moves or interrogate top moves by action plane.TutorialCAZ is not, nor ever intends to be a top chess engine*. Rather, it is a showcase in AI techniques and my attempt to reproduce Deepmind's work. If you'd like to know more about the inner working of self-play reinforcement learning, check out the tutorials. They contain beautifully rendered visualisations to help you understand the way the project works. The tutorials have been organised into four sections:MCTS Demowhich showcases MCTS including the UCB formula. You can render search trees to see it in actionConvolutional Layerswhich shows how convolutional layers work and can combine to transform the 'images' of the gameAlpha Zerowhere you can see CatchemAlphaZero's policies and play against it within an IPython settingChess - which illustrates the exact details of the chess setup. You can interrogate the network and play againt it here (although GUI app preferred)* However, I am a glutton for punishment and will probably rewrite this project in C++ to make it stronger at some point...Thanks for making it this far!CatchemAL