Data-Driven Evaluation of Football Players’ Skills | by Ofir Magdaci | Towards Data Science

By Ofir Magdaci, Data Scientist at Wix.com
To truly grasp all of the concepts to be presented, a familiarity with the StatsBomb xG metric is highly recommended. This great fbref.com article can provide all the information you need.
In the last two posts, I both presented and explained the two main models which allowed us to mathematically represent the language of football: Action2Vec and Player2Vec. We studied their meaning, semantics, and dynamics. We concluded that these models describe what players do, rather than how well they do it. Clearly, players’ skills levels are a missing element in our modeling, and that is what this post is all about.
Football involves a variety of skills. Some are on-ball skills, such as passing and dribbling, while others are off-ball, such as moving into dead zones or marking. The second category is usually more challenging and often requires richer data. So far, on-ball skills have been the most prevalent and thoroughly considered.
The rise of the football analytics industry resulted in an endless number of ways to measure, compare and visualize players’ skills. However, many of the methods are biased due to the team’s tactics or playing style, and generally ignore variations of the action. These raise the need for objective, data-driven measuring that will set the standard for player evaluation.
In this work, I propose a skill evaluating methodology which is on par with existing methods in data science and machine learning. This approach is applicable for any skill, either on-ball or off-ball with suitable data. Here I will analyze and demonstrate this mechanism for shooting.
As in the previous works, the data for this work is based on the Statsbomb open dataset. Documentation is available on the dataset’s Github repository site.
What constitutes a skill may be trickier to define than you imagine. Dictionaries suggest “...the ability to perform a certain action”. However shooting, for example, is a complex task with many variations and difficulties. Additionally, shots can be executed with different techniques, angles, ranges, etc. But ultimately, it remains conjectural — even a top scorer can miss a good chance on any given day.
Think of sporting events like the high-jump. The challenge is significantly less complex — the higher the bar, the more difficult it is to get over it — yet, the outcome is always a probabilistic event for a fixed bar height.
A better definition for a skill, therefore, may be the likelihood to complete a specific task given a specific difficulty. We can describe a player’s skill for completing action “a”, using a probability distribution: Sa(L), where L is the level of difficulty. Given the difficulty of the task, there is an unknown probability, Sa(L=l) of success.
To empirically estimate the distribution of a player’s shooting skill, two things are needed: the expected probability of success (equaling 1 — difficulty) and the outcome for a variety of conditions. For shooting, it corresponds to the xG value of the shot and its outcome.
Estimating the probability of each shot is not a trivial task. Fortunately, Statsbomb was kind enough to add their xG metric to the dataset, which gives us just that — the probability to score from a shooting event. Consequently, the higher the probability of scoring, the lower the shot difficulty rating will be.
N.B.: I have to clarify that the xG model, or any other probability model I will discuss, aims to predict the expected outcome of the shot. Thus, such models ignore personal data, (e.g. the player’s weak foot quality). It is about estimating the ‘net worth’ of each shot rather than its exact outcome.
Once we have estimated the skill distribution, Sa(L), we can compute some metrics on it.
The notion of Lift is broadly used in the data science community. It compares a metric with a baseline, by dividing the two. This ratio indicates how much better the calculated value is over the baseline.
To produce a single number to work with as a KPI (Key Performance Indicator), we can compute the skill distribution AUC (Area Under the Curve). Nevertheless, I find it more informative and communicative to work with the ratio, i.e., the lift, between the AUC and the expected scoring AUC (equaling 0.5), indicating how much better the player is than expected.
Figure 2 shows the xG distribution of several players, contrasted with the expected performance according to StatsBomb’s xG metric.
When described as a distribution, we can see the bigger picture more clearly. We can observe and note differences in performance over levels, detecting mental signals such as missing big chances (e.g., Luis Suarez), and better understand the trade-offs between players. Unsurprisingly, Messi beats the benchmark for any given difficulty, achieving an xG Lift AUC of 1.15 overall.
This collection of KPIs shows how much better the player is in executing a specific shot type compared to the expected performance estimated by the StatsBomb’s xG metric. Its calculation is very straightforward:
There are two main challenges to using this approach. First, estimating difficulties for actions usually requires very rich data, such as Statsbomb 360 or StatsPerform tracking data. Secondly, calculating the probability is not easy, and sometimes is also unclear. For example, how does one can estimate the difficulty of a dribble? What may work against one opponent could be completely ineffective against a different player with superior intercept skills.
To compensate for the hurdles, the metric offers an abundance of advantages:
✓ It is simple and easy to implement.
✓ It is at par with existing measures in data science — Lifts and AUC.
✓ Intuitive bottom line: how much better the player is than expected.
✓ It is robust and can be applied to any sport and skill.
But perhaps the biggest added value is the ability to only evaluate different types of shots separately. For example, we can calculate the xG Lift only on long-range shots, free-kicks, or one-on-one shots. As you probably already know, I rarely advance ideas without first executing them myself...
Figure 3 shows the skills radar chart for Neymar JR as estimated from the StatsBomb open dataset. The values are the percentiles of the Lifts, considering all eligible players in the dataset — those players with at least 30 shots.
This gives us a clear, consistent, standardized, and normalized scale. Alternatively, when all categories are measured by Lifts, the raw Lifts values can be used as well.
Similar categories are placed as closely together as possible to avoid ordering bias. The player shooting skill is divided into different sub-categories and contexts. These contexts can be applied for performance evaluation, player search, or development. Usually, players excel in some of the shot types while struggling with others, unless, of course, your name is Leo Messi (Figure 4).
Looking closer, the xG Lift plot resembles the radar chart. In fact, it is a reduction of the radar chart, where axes are collapsed to a single shooting index. High xG values certainly contain sub-skills, such as one-on-one finishing, penalties, and part of the headers. Low xG values, however, cover mostly long-shots and free-kicks. In both, however, the higher the overall volume, the higher the recorded skill level is. This is another hint for how to properly order the categories.
Of course, there is both good news and bad news with this technique... The good news is that we can apply this method to any skill we can measure, such as dribbling and passing! However, this also includes bad news, since we can’t measure some other skills using events data (e.g. dead zone coverage or marking).
The Statsbomb open dataset is essentially a collage of selected samples of thousands of matches over two decades. These matches originate from many competitions, continents, and both genders. More importantly, it is NOT randomly collected (see: the docs). As a result, the data is very sparse, biased, and tends to focus on the top teams and leagues rather than on the true distribution of football players and matches.
Consequently, the distribution of shots per player was so skewed that I wasn’t able to plot it properly. I would rather show it to you in numbers:
Mean shots per player: 5.2, STD: 46.4.  ╔════════════╦══════════════╗  ║ Percentile ║    #Shots    ║  ╠════════════╬══════════════╣  ║    25%     ║        0     ║  ║    50%     ║        1     ║  ║    75%     ║        1     ║  ║    90%     ║        8     ║  ║    95%     ║       14     ║  ║    99%     ║       56     ║  ║    Max     ║    2,190     ║ Messi  ╚════════════╩══════════════╝
Since the dataset is not random, most of the players have very few shots assessed. Addressing this, I filtered out all players who had less than 30 shots — yet some Lifts are still too radical (Figure 5). For players with hundreds of shots, we do get accurate, realistic numbers.
The radar chart is probably the most favorable chart among the football analytics community, as they can present many skills, on the same canvas, in a very intuitive way. Unfortunately, it has many possible failure points, too — likely making it the most misused chart of all.
Following is the recipe to help you create a flawless radar chart:
1. Watch the order of your variables: the radar chart logic is simple — the bigger the volume of the polygon, the better the player is in the listed skills. The volume is created by connecting adjacent dots. Thus, switching the positions of even two skills in the chart might cause a dramatic effect:
2. Avoid using tactic-dependent categories — this is probably the biggest and most common mistake of all. There is no point in counting the number of successful dribbles/tackles or the aerial battles won during a match. These measures are highly dependent on the team quality, style of play, or the manager’s instructions. Instead, use metrics such as “% of successful defensive one-on-one actions”.
3. Use relative numbers — all values should be relative to a meaningful benchmark. For example, instead of presenting the “% of dribbles a player won within a match”, we can use percentiles.
4. Keep your scale fixed — such as when using Lifts or percentiles. When each dimension scale is different, we might mislead the reader by producing biased charts.
5. Add a baseline — for example, the average performance of players of the same position — this is essential for global comparability against other players in the dataset. It is even more crucial when the scale is not consistent.
These analyses require a bigger and more balanced dataset. However, the Football2Vec library also includes their code for you to explore:
Comparing xG Lift curve over dimensions — this analysis may reveal some insights regarding changes in skills level across different playing positions, ages, and competitions.
Comparing training Lifts VS. matches Lifts — This can be an indicator of mental or tactical effects that influence the player’s performance.
It’s time for the part where all things come together with a beautiful Streamlit UI. This super easy-to-use Python package has become my favorite for interacting with my work and making it welcoming for others. To this end, I built a simple player dashboard with the following features:
Analyzing the player’s skills and performance over seasons. Contain two Plotly animated charts:
All code is available via the Football2Vec Python open-source library.
In this post, we unraveled the world of skill evaluation. Starting from zero, we first defined what a skill is, how we can mathematically formalize it, and how we can empirically measure it; we got to know the notions of Lift (Formula 1), the xG Lift, and the xG Lift AUC — for evaluating shooting across its different types.
Then, we shifted our focus to how to properly visualize skills and emphasize the important best practices; and with the power of Streamlit, we were able to translate complex models and statistics into an understandable interactive beautiful UI.
Throughout these three posts, we went through an end-to-end data science journey to create a holistic representation of the language of football. Our goal is yet to be accomplished, but in the continuing quest for achieving it, we’ve just made it through to base camp.
Now our methodology covers both what the player does, and how well he does it (at least how well he shoots). All that remains is to find a downstream task with a sufficient amount of data to learn from. And, of course, additional skills can be modeled as needed.
I hoped you found this trilogy interesting to read. Thank you for all the feedback, it was truly fun!
Game, set, and match.