Data Science, Chess, and Modeling
By: Brayton Hall
Tags: Data Science, Modeling, Philosophy, Chess

Data Science, Chess, and Modeling

How Chess Can Improve Your Data Science Skills

Photo by George Hodan, Public Domain

Chess and data science have a lot in common. Some seemingly surface-level parallels include imposter syndrome and a feeling of powerlessness in the face of overwhelming complexity and indecision, all on top of a time crunch.

If we look closer, though, these upfront similarities belie truly deeper parallels between fields. These types of experience don’t often come from light-hearted hobbies: frustration, complexity, despair, ecstasy, confusion, pride, and understanding are the earmarks of a pursuit which wholeheartedly consumes the practitioner for the duration of the activity and beyond, and the lifetimes and books devoted the study of chess or data science demonstrates the intensity of experience which belongs to both.

Of course, much of what we call ‘data science’ has historically been plain old statistics, though admittedly without the furor of machine learning or deep learning, and experienced data scientists know all too well that often A/B testing, or two-sample hypothesis testing, can provide far more value and intelligible insight than neural networks.

This is a good entry point into the main fundamental parallel between chess and data science, which is that in either category, experience, pattern recognition, and intuition are driving factors of success. It is fairly well-established that chess grandmasters, in particular, are successful by virtue of their deep pattern-recognition attained through elongated and intensive experience. While the exact explanatory aspect of pattern recognition is unclear (since young grandmasters with only a couple of decades of experience can win over older grandmasters with lifetimes of experience), it is clear that chess is strangely more Bayesian than it may initially appear, especially in the context of falsely algorithmic attitudes surrounding chess skill.

The Primary Skill Behind Chess

A fascinating discussion of chess grandmaster thinking can be found here. While pattern recognition is a general theme, the primary focus of the discussion concerns the thinking process of grandmasters itself. What are the typical thoughts and observations that drive a peak-operating grandmaster to burn up to 6,000 calories a day during a tournament?

Robert Sapolsky, who studies stress in primates at Stanford University, says a chess player can burn up to 6,000 calories a day while playing in a tournament, three times what an average person consumes in a day. Based on breathing rates (which triple during competition), blood pressure (which elevates) and muscle contractions before, during and after major tournaments, Sapolsky suggests that grandmasters’ stress responses to chess are on par with what elite athletes experience. -Aishwarya Kumar

The answer is not deep algorithmic thinking. There is a myth that chess grandmasters are human computers, somehow capable of multi-thread processing and handling superhuman mental permutations at a scale far higher than other humans.

Even supercomputers haven’t solved chess, given that the mere lower bound of the number of possible chess games is described by 10¹²⁰, which is the game-tree complexity of chess also known as the Shannon Number. Even looking 6 or 7 moves ‘into the future’ is just as difficult for a chess grandmaster as it is for a novice in terms of permutation processing.

The consensus, rather, is that grandmasters are far more quick at seeing threats, summarizing positions and holding it in long-term memory, and recognizing the inevitable lines from those states.

While this seems to imply that grandmasters are seeing ‘more into the future’, a novice can equally see checkmate in 4 or 5 moves once the pieces on the board are reduced to two kings and pawn approaching queen promotion. The difference is that grandmasters can handle more complexity and understand the most important information amid the noise, and recognize the relatively few (and often forced) options before them, effectively reducing the possible permutations to a surprisingly small decision tree. The important point is that they aren’t better permutation processors: they are just more capable of simplifying a complex state to less complex one, and then making their decisions based on prior experience with game states and pattern recognition, along with a supreme mental representation of the state.

Garry Kasparov could supposedly remember ‘the moves of all the games he had played in the past 6 months’, while Magnus Carlsen supposedly memorized 10,000 chess games, in an article by Bill Wall which seems to confirm the explanatory significance of long-term memory in chess.

How does any of this relate to data science?

Because in data science, much of this process would be called modeling! The vocabulary of STEM can often be confining as it is useful, and while modeling has specific connotations with statistical models, the process of model-building is nearly identical.

While we often think of a linear regression or random-forest model in Python, built with sklearn and fit on a set of numpy arrays, the actual process of feature engineering, log-transforming, cleaning and imputing null values based on context, and visualizing data for deeper understanding of the data all utilize the same skills, for the same purpose, as a grandmaster staring at the board for an hour blazing away calories: to improve prediction and to gain a larger scope understanding of the data on an intuitive level. This larger-scope is often referred to as ‘domain knowledge’ or ‘business understanding’, or theoretical understanding in a scientific context, and fits into the cross-industry standard process for data mining, or CRISP-DM:

CRISP-DM by Kenneth Jensen, Creative Commons

Except that in chess, we might describe this first step as ‘game state understanding’, a deep and intuitive understanding of color-respective proximity to checkmate, or advantage, which may be roughly measured by average centipawn loss, or computer-determined strength of play. The ‘deployment’ step in CRISP-DM would be analogous to an actual chess move.

While statistical modeling misses the connotations of a chess grandmaster’s thinking process, the intent and flow of steps are shockingly similar, particularly in how the modeling ‘fit’ and ‘predict’ steps are equivalent to the ‘looking 20 steps ahead’ type of algorithmic processing grandmasters are so stereotypically known for, when in fact much of the real skill comes from longer-term memory, pattern recognition, contextual understanding of the problem, and a supreme agility and precision of mental modeling.

I believe chess can be a worthy application of time for an aspiring data scientist, in that it’s both personally rewarding and enriching for the skillset of a data scientist.

Also, it’s fun. Thanks for reading!