FIFA Ultimate Team Machine Learning Project | by Jean-Paul Zambrano-Leon | Towards Data Science

This project follows the principles/steps outlined in the Machine Learning End-To-End guideline by ML consultant Aurélien Géron.
Machine Learning (ML) is a subset of Artificial Intelligence (AI) that uses computer algorithms to find patterns in large datasets without being explicitly programmed. ML gives the computer the ability to learn and improve on its own so that it can produce the best model from the data. A lot of applications nowadays use ML such as:
Product Recommendations
Email Spam Detector
Malware Detection
Face Recognition
I’ve been playing the FIFA games since I can remember. I love being able to play as my favorite football players and teams. Although the games nowadays get a bad rap for being mostly pay-to-win, I still find myself enjoying the spoils of FIFA Ultimate Team (FUT) and the competitive multiplayer. With that being said, I’ve always wondered how the developers decide on player ratings, and then I asked myself, “Could a machine do it too?”. It could very well be that the developing team at Electronic Arts (EA) use an ML model to decide the player ratings, but that didn’t stop me from building my very own. This project was my attempt at creating an ML model that can predict the overall rating of any player.
I used Google Colabs to complete this project.
I found a large dataset of the FIFA games on Kaggle. The dataset consists of data from FIFA 15 to FIFA 20. We will be using the latest FIFA game in the dataset which is FIFA 20. The dataset can be downloaded from Kaggle. The dataset consists of 18,278 rows and 104 columns. The columns contain features such as Player Name, Age, Height, Weight, Nationality, and many more. Since we are trying to predict the overall rating, we would only need to focus on the features that have correlations with the overall rating, but we’ll get to that later.
The first thing that any ML Engineer should do is to define the problem. For this project, the problem we are facing is accurately predicting the player ratings without having to hard code an algorithm, so building a workable ML model should be our end goal. In FIFA 20, a player’s overall score is defined by the score of each player attribute such as Pace, Shooting, Passing, Dribbling, and many others. We are trying to predict the overall score based on the player’s offensive and defensive attributes. The data already comes with labeled training examples which means we are dealing with a supervised learning problem. It is also a typical regression task since we are tasked with predicting a value. More specifically, this is a multiple regression problem since the model will use multiple features to make a prediction.
Now that we have defined the problem, we need to select a performance measure. I like to use the Root Mean Square Error (RMSE). This is the preferred performance measure for regression problems. The RMSE is a measure of how well the model performed. It gives an idea of how much error the system makes in its prediction. RMSE is the calculation of the residuals (prediction errors) with respect to the line of best fit. The formula can be seen below:
To learn more about RMSE and how it works, I recommend reading this article.
Now we are ready to start coding!!!
The first thing that we need to do is to import the most common data science libraries such as numpy, scipy, and pandas. We also need to import matplotlib and seaborn so that we can plot graphs to help us visualize the data.
# Common importsimport numpy as npimport pandas as pdimport osimport seaborn as snsfrom scipy import statsimport missingno as msno# To plot pretty figures%matplotlib inlineimport matplotlib as mplimport matplotlib.pyplot as pltfrom matplotlib import stylesns.set(style='ticks', color_codes=True)sns.set(style='darkgrid')import plotly.express as px# Ignore useless warnings (see SciPy issue #5998)import warningswarnings.filterwarnings(action="ignore", message="^internal gelsd")warnings.filterwarnings('always')warnings.filterwarnings('ignore')
I saved the dataset to my GitHub repository which makes it easier for me to download the dataset without having to search for it on my machine. Having a function to download the data online is really useful if the data is constantly changing. It also saves the trouble of having to install the dataset on multiple machines.
import urllib.requestDOWNLOAD_ROOT = "https://raw.githubusercontent.com/jpzambranoleon/ML_Projects/master/"TRAIN_PATH = os.path.join("datasets", "players_20")TRAIN_URL = DOWNLOAD_ROOT + "datasets/players_20.csv"def fetch_train_data(train_url=TRAIN_URL, train_path=TRAIN_PATH):  os.makedirs(train_path, exist_ok=True)  csv_path = os.path.join(train_path, "players_20.csv")  urllib.request.urlretrieve(train_url, csv_path)
The fetch_train_data() function is used to fetch the dataset.
fetch_train_data()
The dataset is downloaded and saved in the files section of Google Colabs under the folders datasets and players_20. Now all that’s left to do is to load the data into a pandas DataFrame. Luckily, there is a function that we can create to return a pandas DataFrame object.
import pandas as pddef load_train_data(train_path=TRAIN_PATH):  csv_path = os.path.join(train_path, "players_20.csv")  return pd.read_csv(csv_path)
I called the load_train_data() function into player_data variable to save the DataFrame into a variable to make things easier. The DataFrame’s head() method is used to look at the top five rows of the data.
player_data = load_train_data()player_data.head()
I used the shape method to return a tuple representing the dimensionality of the DataFrame. By using the shape method, we can see how many rows and columns there are in the dataset.
player_data.shape(18278, 104)
As we can see from the tuple, the dataset has 18,278 rows and 104 columns. We don’t need to use all the columns so it’s best if we truncate the DataFrame to only include the target column and the player stats columns. There are 34 player stats columns so we’ll need to select those.
#player stat featurescolumns = ['overall','attacking_crossing', 'attacking_finishing', 'attacking_heading_accuracy', 'attacking_short_passing', 'attacking_volleys', 'skill_dribbling', 'skill_curve', 'skill_fk_accuracy', 'skill_long_passing', 'skill_ball_control', 'movement_acceleration', 'movement_sprint_speed', 'movement_agility', 'movement_reactions', 'movement_balance', 'power_shot_power', 'power_jumping', 'power_stamina', 'power_strength', 'power_long_shots', 'mentality_aggression', 'mentality_interceptions', 'mentality_positioning', 'mentality_vision', 'mentality_penalties', 'mentality_composure', 'defending_marking', 'defending_standing_tackle', 'defending_sliding_tackle', 'goalkeeping_diving', 'goalkeeping_handling', 'goalkeeping_kicking', 'goalkeeping_positioning', 'goalkeeping_reflexes']player_df = player_data[columns]player_df.head()
I used the info() method on the new DataFrame to get a quick description o the data.
player_df.info()
We can see from the data that there are 18,278 non-null values in the DataFrame which means we don’t have to worry about missing values.
To get a statistical description of the data, we can use the describe() method.
player_df.describe()
Just for good measure, let’s use the shape method again to see the dimensionality of the new DataFrame.
player_df.shape(18278, 35)
Another great way to get a feel of the data is to plot a histogram of every numerical attribute. To plot a histogram of every attribute, just call the hist() method on the DataFrame. The hist() method requires the matplotlib library. You can specify the number of bins for each histogram and you can also specify the size of the figure. I chose to go with 40 bins and a figure size of (27,17), but all of that is completely up to you. Histograms are useful because they show the number of instances (y-axis) that have a given value range (x-axis).
player_df.hist(bins=40, figsize=(27,17))plt.show()
As we can see from the figure, each histogram has a value range of 0 to 100. This is a good indication that every attribute has the same scale which means we don’t have to do any feature scaling transformations.
For good practice, I like to be completely sure that there are no missing values in the data, so I use the isna().any() method to return a True or False statement on whether or not the data has missing values.
player_df.isna().any()
As we can see from the output, the return value is False on every attribute which means that there are no missing values. We are ready to continue!!!
After exploring the data, the next thing we need to do is to split the data into training and testing sets. We can do this by splitting the data ourselves, or by creating a function to do it for us. Luckily for us, Scikit-Learn has a cool feature that can split the dataset into training and testing sets. The train_test_split() function can easily split the data into the sets we need. I set test_size=0.2 so that 20% of the data is stored in the test set. The other 80% of the data is stored in the training set. The training set should have more of the data because it is the set which the model learns from. The testing set will be the set that we use to validate the model.
from sklearn.model_selection import train_test_splittrain_set, test_set = train_test_split(player_df, test_size=0.2, random_state=42)print("Length of training data:", len(train_set))print("Length of testing data:", len(test_set))print("Length of total data:", len(player_df))Length of training data: 14622 Length of testing data: 3656 Length of total data: 18278
You can compute the standard correlation coefficient between each pair of attributes using the corr() method. Looking at all the correlations between each attribute will take up too much time, so let’s just see how much each attribute correlates with the overall value.
fifa_df = train_set.copy()corr_matrix = fifa_df.corr()corr_matrix['overall'].sort_values(ascending=False)
The correlation coefficient ranges from -1 to 1. When the correlation coefficient is close to 1, it means that there is a strong positive correlation; for example, overall tends to go up when movement reactions goes up. When the correlation coefficient is close to -1, it means that there is a strong negative correlation (i.e., the opposite of a strong positive correlation).
To visualize the correlation between attributes, we can use the pandas scatter_matrix() function. This function plots every numerical attribute against every other numerical attributes. Let’s just look at the top five promising attributes to make things easier since there are 35 numerical attributes.
from pandas.plotting import scatter_matrixattributes = [‘overall’, ‘movement_reactions’, ‘mentality_composure’, ‘power_shot_power’, ‘mentality_vision’]scatter_matrix(fifa_df[attributes], figsize=(15,12))plt.show()
Strong correlation relationships tend to be linear, so the most promising attribute to predict overall is movement reactions. As we can see from the figure, movement reactions and overall have a very strong linear relationship.
Let’s take a closer look at the correlation scatterplot.
fifa_df.plot(kind=’scatter’, x=’movement_reactions’, y=’overall’, alpha=0.1, color=’red’)plt.show()
The correlation is indeed very strong; you can clearly see an upward trend.
Now, it’s time to prepare the data for ML algorithms.
I split the training set and testing set into separate sets of features and targets. The DataFrame y_train and y_test contain the target values (the target value is overall) and X_train and X_test contain the feature values (every other attribute that correlates to the target value).
y_train = train_set['overall']X_train = train_set.drop('overall', axis=1)y_test = test_set['overall']X_test = test_set.drop('overall', axis=1)
Let’s first train a Linear Regression model using Scikit-Learn’s LinearRegression() function:
from sklearn.linear_model import LinearRegressionlin_reg = LinearRegression()lin_reg.fit(X_train, y_train)
Let’s measure the regression model’s RMSE on the whole training set using Scikit-Learn’s mean_square_error() function:
from sklearn.metrics import mean_squared_errory_predictions = lin_reg.predict(X_train)lin_mse = mean_squared_error(y_train, y_predictions)lin_rmse = np.sqrt(lin_mse)lin_rmse2.493574884183225
Not a bad score!!! It seems that a Linear Regression model could do the trick. From what we saw from the scatter matrix is that not every attribute has a linear relationship. For nonlinear relationships, we would need a more powerful model so let’s train a Decision Tree Regression model. This is a powerful model, capable of finding complex nonlinear relationships in the data. Let’s use Scikit-Learn’s DecisionTreeRegressor() function:
from sklearn.tree import DecisionTreeRegressortree_reg = DecisionTreeRegressor()atree_reg.fit(X_train, y_train)
Again, let’s measure the model’s RMSE:
y_predictions = tree_reg.predict(X_train)tree_mse = mean_squared_error(y_train, y_predictions)tree_rmse = np.sqrt(tree_mse)tree_rmse0.0
What?! An RMSE of 0.0!!! Is this a perfect model?
Actually, it could be that this model has badly overfit the data. That’s why in every ML project it is important that you perform model validation, but we’ll get to that later.
For now, let’s just train our final model which is a Random Forest Regression model.
We can train a Random Forest Regression model using Scikit-Learn’s RandomForestRegressor() function:
from sklearn.ensemble import RandomForestRegressorforest_reg = RandomForestRegressor()forest_reg.fit(X_train, y_train)
Let’s look at the RMSE for this model:
y_predictions = forest_reg.predict(X_train)forest_mse = mean_squared_error(y_train, y_predictions)forest_rmse = np.sqrt(forest_mse)forest_rmse0.4774070457668509
It’s a better score than our Linear Regression model, but it’s not as good as our Decision Tree model. Again, it could be that our Decision Tree model has badly overfit the data, which is why we always need to validate each model before selecting the best one. An easy way to validate a model’s accuracy is to use Scikit-Learn’s K-fold cross-validation feature. Cross-validation randomly splits the training set into 10 distinct subsets called folds, then it trains and evaluates the models 10 times, picking a different fold for evaluation every time and training on the 9 folds. The result is an array containing the 10 evaluation scores. We need to take a look at the mean validation score before we can choose the best model.
Let’s look at the cross-validation scores of our Linear Regression model:
from sklearn.model_selection import cross_val_scorescores = cross_val_score(lin_reg, X_train, y_train,scoring='neg_mean_squared_error', cv=10)lin_reg_scores = np.sqrt(-scores)def display_scores(scores):  print("Scores:", scores)  print("Mean:", scores.mean())  print("Standard Deviation:", scores.std())display_scores(lin_reg_scores)Scores: [2.48985397 2.509888   2.50910312 2.4642167  2.46189444 2.51445854  2.48370267 2.4734443  2.48382446 2.59475861] Mean: 2.4985144826450054 Standard Deviation: 0.03662607159427437
Now let’s look at the cross-validation scores of our Decision Tree model:
scores = cross_val_score(tree_reg, X_train, y_train,scoring='neg_mean_squared_error', cv=10)tree_scores = np.sqrt(-scores)display_scores(tree_scores)Scores: [2.03976196 2.10100389 2.24933491 2.18751588 2.13289653 2.1158318  2.10529916 2.08391009 2.30935171 2.15395796] Mean: 2.147886387907302 Standard Deviation: 0.07682311253910523
Now that’s a more believable score than the first. It’s still better than our Linear Regression model, but let’s look at the cross-validation score of our Random Forest model:
scores = cross_val_score(forest_reg, X_train, y_train,scoring='neg_mean_squared_error', cv=10)forest_scores = np.sqrt(-scores)display_scores(forest_scores)Scores: [1.29156727 1.24883845 1.2625419  1.288613   1.22436926 1.30612055  1.25621857 1.24755531 1.30399247 1.28656714] Mean: 1.2716383911151843 Standard Deviation: 0.026074298378757566
Wow!!! This might just be the best score we’ve seen. Random Forests look very promising. This is the model that we’ll use for our project.
Now that we have selected our model, we need to fine tune it. Fine tuning requires that we fiddle with the hyperparameters manually, until we find a great combination of hyperparameter values. Fine tuning is very tedious work and can take a long time if the dataset is large. You may not even have time to explore many combinations. Instead of doing everything manually, there is an easier option that Scikit-Learn provides.
Instead of fiddling with the hyperparameters manually, you should use Scikit-Learn’s GridSearchCV to search for you. All you need to do is tell it which hyperparameters you want it to experiment with and what to try out, and it will use cross-validation to evaluate all the possible combinations of hyperparameter values. The code below provides the parameters that I used to experiment with.
from sklearn.model_selection import GridSearchCVparam_grid = [{'n_estimators': [3, 10, 30], 'max_features': [2, 4, 6, 8]},{'bootstrap': [False], 'n_estimators': [3,10], 'max_features': [2,3,4]},]forest_reg = RandomForestRegressor()grid_search = GridSearchCV(forest_reg, param_grid, cv=5,scoring='neg_mean_squared_error',return_train_score=True)grid_search.fit(X_train, y_train)
When it’s done training, you can get the best combination of parameters like this:
grid_search.best_params_{'max_features': 8, 'n_estimators': 30}
You can also get the best estimator directly:
grid_search.best_estimator_RandomForestRegressor(bootstrap=True, ccp_alpha=0.0, criterion='mse', max_depth=None, max_features=8, max_leaf_nodes=None, max_samples=None, min_impurity_decrease=0.0,                 min_impurity_split=None, min_samples_leaf=1,                     min_samples_split=2, min_weight_fraction_leaf=0.0,                       n_estimators=30, n_jobs=None, oob_score=False,                       random_state=None, verbose=0, warm_start=False)
Of course, you can experiment with any hyperparameter values, but I think this will work for now.
After tweaking the model on the training set, it’s finally time to evaluate the final model on the test set. The process is pretty similar, but this time we need to specify the best estimator to the final model. Again, we’ll need to measure the performance of the final model. The code to do that is below:
final_model = grid_search.best_estimator_final_predictions = final_model.predict(X_test)final_mse = mean_squared_error(y_test, final_predictions)final_rmse = np.sqrt(final_mse)final_rmse1.135337798136083
The score is the best we have seen so far, not to mention it was also used on data that the model has not previously seen. If I’m being honest, I didn’t expect the model to do this well, but we can’t just assume that the RMSE will always be a certain value. We want to have an idea of how percise this estimate is. For this, you can compute a 95% confidence interval for the generalization error using scipy.stats.t.interval():
confidence = 0.95squared_errors = (final_predictions - y_test)**2np.sqrt(stats.t.interval(confidence, len(squared_errors)-1,loc=squared_errors.mean(),scale=stats.sem(squared_errors)))array([1.10181563, 1.16789818])
We can see that the RMSE will be between 1.10 and 1.17
95% of the time.
We are pretty much done with the model, but just out of curiosity, we would want to see how the predictions differed from the actual target. To do this, just selected five instances from the test set and use the predict() method on the final model:
some_data = X_test.iloc[:5]some_label = y_test.iloc[:5]print("Predictions:", final_model.predict(some_data))print("Labels:", list(some_label))Predictions: [64.2    73.8     68.76666667 68.13333333 63.4     ] Labels: [64, 74, 69, 68, 63]
The predictions are pretty accurate. I’m surprised that the model performed this well. I’m very satisfied with the results.
This project showed us that we can build an ML Model to predict a player’s overall FUT rating. This project was a lot of fun to make and I learned a lot from it. I’m passionate about ML and all the possibilities that it has to offer. I am very excited to explore more of this field and learn everything that I can along the way. I hope you enjoyed this article and maybe you can start your own ML project someday!!!