Hand-made ML vs. AutoML — Who will win? | by Sebastian Zolg 🤝 | Towards Data Science

As part of the Microsoft Power Platform, AI Builder can create Machine Learning Models with no code. This approach is often referred to as AutoML.
But how good are those models compared to a hand-made model written in Python?
We’ll compare various models with different complexity against each other and examine the results in a final comparison.
Who should read this? This story is written for everyone interested in Machine Learning and No-code Platforms. The data scientists among us will understand what AutoML is capable of and how it looks and feels. For everyone else, it gives a good understanding of what to expect from such a model, comparing it to a hand-made one. I’ll show the most important code-snippets, so everyone has the chance to learn what it takes to build such a model, and better to understand the other side of the same coin.
This article isn’t a step-by-step guide for model creation in AI Builder. If you want to learn how to create a AI Builder model from scratch, just follow my other story here.
We’ll use the popular Ames Housing Prices dataset from Kaggle. The Dataset contains information from the Ames Assessor’s Office used in computing sale prices for individual residential properties sold in Ames, IA from 2006 to 2010. It covers information like the overall quality of a house, year built, the lot size, garage cars, floor sizes, kitchen quality, and many more. Of course, it holds the actual sale price per property and is perfectly suited for a predictive machine learning model.
Our goal is to build a predictive machine learning model based on the above dataset. We aim to predict the Sale Price of new residential properties poping in. We assume a linear relationship between most of the information in the dataset, so we’ll approach this exercise using various linear regression models to predict a future sale price.
We’ll compare models of different complexity, both for a hand-made ML model and the low-code approach with Microsoft's AI Builder.
The raw house prices dataset covers 2430 records with 82 features (each house's properties), which we first split into a sample and a validation dataset with an 80/20 split.
>>> data_df.shape(2430, 82)
The sample data is used for training and testing all models during creation.
The validation data is used as completely new and unseen data during our final evaluation phase. It will show us how well our model generalizes and performs on new data. Furthermore, we make sure that the AI Builder models' reported results aren’t tweaked in a way.
Remember, AutoML is mostly a black box, and we should treat the reported numbers with caution until we have established enough trust, e.g., by letting it work on new data.
# Code to split the data 80/20import pandas as pdsample_data_df = data_df.sample(frac=0.8,                                random_state=1).copy()validation_data_df =     data_df.loc[~data_df.index.isin(sample_data_df.index)].copy()
After splitting the data, we have two batches to work with. Our sample set contains 1944 records, while our validation set contains 486 data points.
>>> sample_data_df.shape(1944, 82)>>> validation_data_df.shape(486, 82)
Before moving on, we’ll export the datasets back into CSV to import them into Microsoft’s Power Platform.
# Function to wrap a simple CSV export using Pandas.to_csv()export_to_ai_builder(sample_data_df,                      validation_data_df,                      description='raw')
Exploratory Data Analysis (EDA), filtering outliers, or adding missing values are an essential part of ML model creation. It’s all about understanding and preparing the data. Further, enriching the data with additional features is an important job to increase any model's performance. This includes expressing existing attributes as new features or representing existing features as polynomials on a different scale. This process is called feature engineering.
We skipped all of this for our AI Builder models and just threw the data into the AutoML engine to let it do its work.
For the hand-made model, we did a couple of preparation steps as follows:
We added polynomial features of various degrees for all continuous variables.
# Add polynomial featuresdef add_polynomial_features(data_df,continuous_columns):    data_df = data_df.copy()    for c in continuous_columns:        for d in [0.5, 2, 3]:            name = '{}**{}'.format(c, d)            data_df[name] = data_df[c]**d        return data_df
We added a couple of intuitive new features derived from existing ordinal and discrete features.
- Is old House - Is new House- Is a remodeled House- Has a pool
Another important step was filtering outliers, which we identified during the EDA phase. For instance, we saw a few houses with a living area above 4000 square feet and a corresponding high sale price, which we filtered out.
cleaned_df = cleaned_df[cleaned_df['Gr Liv Area'] < 4000]
How do we compare the performance of the models?
Well, we choose Mean Absolute Error (MAE) as our metric of choice. We could use many other metrics, but this one is most human-readable and easy to interpret.
In very simple words, we measure the “average” deviation between the predicted sale price and the actual sale price, expressed in USD$.
This tells us how far away, on average, our prediction is from the actual sale price. The lower the number, the better our prediction.
We’ll compare models with different levels of complexity in terms of input data (features) used. Remember that our dataset covers 82 features (properties of houses), with various impacts on the Sale Price.
We’ll compare every model against our baseline. Our baseline is simply the mean average of all house prices as our “best guess.” As you can see from the plot below, estimating a house's sale price by taking the mean average doesn’t get us good results. In fact, the baseline MAE on the sample dataset is 58,033$ “off.” We can do much better using Machine Learning, either hand-made or with AutoML.
Our simple model uses only 4 features selected by pure intuition. We use the same features for hand-made and AI Builder models.
- Year Remod/Add- Lot Area- Overall Qual- Exter Qual
Our simple model in AI Builder reports a B-rating measured with R-squared. It’s interesting to note that the Overall Quality is the most influential feature according to AI Builder’s dashboard. Intuitively, this makes a lot of sense.
For our hand-made model, we use a simple linear regression model from the sklearn library.
from sklearn.linear_model import LinearRegressionlinear_simple = LinearRegression()linear_simple.fit(X_tr, y_tr)y_pred_linear = linear_simple.predict(X_te)mae_simple = MAE(y_te, y_pred_linear)
The reported MAE of this simple model is 27,944$. It’s already twice as good as our baseline.
For our intermediate model, we chose twelve different features. Instead of selecting the features by hand, we use SelectKBest from the sklearn library to automatically select the most influential features.
from sklearn.feature_selection import SelectKBest, f_regression# Create and fit selectorselector = SelectKBest(f_regression, k=12)selector.fit(X, y)# Get columns to keep and create new dataframe with those onlyauto_selected_cols = selector.get_support(indices=True)
Looking at the automatically selected features below, it all makes sense.
- Overall Qual- Year Built- Year Remod/Add- Exter Qual- Bsmt Qual- 1st Flr SF- Gr Liv Area- Full Bath- Kitchen Qual- Fireplace Qu- Garage Finish- Garage Cars
Using those features as the input for the AI Builder intermediate model, we get an A-rating with 87% r-squared. This is way better than the simple model. We can see from the most influential data that our dataset is better balanced than the simple model. However, Overall Quality is still the most important feature, followed by 1st Flr SF and Gr Liv Area.
We use the same Linear Regression model and our 12 auto-selected features for our hand-made model.
from sklearn.linear_model import LinearRegressionlinear_intermediate = LinearRegression()linear_intermediate.fit(X_tr, y_tr)y_pred_linear = linear_intermediate.predict(X_te)mae_intermediate = MAE(y_te, y_pred_linear)
The MAE of this intermediate model is 19,900$. It outperforms the simple model significantly.
We use all of the 82 available features for our final model and let the model itself find the way through this data jungle.
With all the data points as input, the AI Builder model gets an A-rating with 91% r-squared. Overall Quality still dominates the field of the most influential data.
For the hand-made model, we chose a Ridge Regressor from the sklearn library and tweaked its hyperparameters using Grid Search in a slightly more complex setup.
from sklearn.linear_model import Ridgegs_results = []for run_idx in range(10):    X_tr, X_te, y_tr, y_te = train_test_split(        X, y, test_size=0.2, random_state=run_idx)# Standardize features    scaler = StandardScaler()    X_tr_rescaled = scaler.fit_transform(X_tr)    X_te_rescaled = scaler.transform(X_te)# Grid search    for alpha in np.logspace(-1, 4, num=20):        # Create and fit ridge regression        ridge = Ridge(alpha=alpha)        ridge.fit(X_tr_rescaled, y_tr)# Save model and its performance on train/test sets        gs_results.append({            'model': ridge,            'alpha': alpha,            'run_idx': run_idx,            'train_mse':                MSE(y_tr, ridge.predict(X_tr_rescaled)),            'train_mae':                MAE(10**y_tr, 10**ridge.predict(X_tr_rescaled)),            'test_mse':                MSE(y_te, ridge.predict(X_te_rescaled)),            'test_mae':                MAE(10**y_te, 10**ridge.predict(X_te_rescaled))})gs_results = pd.DataFrame(gs_results)
It’s worth the effort! The MAE of this complex model is 14,341$. It’s notably better than the intermediate model and far away from our baseline.
Now, how can we compare the AI Builder’s model performance with its r-squared rating to our hand-made models with its MAE metric?
Luckily, AI Builder offers us to download additional metrics for its models, including MAE.
Just click the three dots on the model dashboard and select Download detailed metrics.
This will download a minified .json file, which you can open and reformat in your favorite editor. Just look up the Mean Absolute Error in the list of available metrics.
Now that we know how to get comparable metrics for both approaches, we can make a final comparison of the results.
In addition to that, we’ll use our validation dataset (remember, this data is unseen by both approaches) to see how well the model generalized, and if the results are stable on new data.
For AI Builder, we import the additional validation data and trigger all three models to predict. We then export the predicted Sale Prices and calculate the MAE “by-hand.”
We do the same for the hand-made model and pass the additional validation data through our pipeline, using the previously fitted estimators.
Here are the final results:
# Results on Sample DataBaseline Model has an MAE of 58,033$ML Simple Model has an MAE of 27,944$ML Intermediate Model has an MAE of 19,900$ML Complex Model has an MAE of 14,340$AI Builder Simple Model has an MAE of 24,596$AI Builder Intermediate Model has an MAE of 17,946$AI Builder Complex Model has an MAE of 14,302$
Looking into the results, we can focus on two major findings.
We can see similar results for the AI Builder models compared to hand-made ML models— all without a single line of code, EDA, or data cleanup!
This is quite impressive, and we could conclude that AutoML democratizes machine learning.
But, does this come with a cost? Of course, it does!
As we can see from the final results, our complex hand-made model shows better results on new and unseen (validation) data.
AI Builder Complex Model has an MAE of 16,405$ML Complex Model has an MAE of 13,938$
Why is this? Throughout creating the hand-made model, we did a lot of EDA upfront to understand the data better. This allowed us to clean the data, e.g., filtering outliers. Furthermore, we engineered a couple of additional features for the complex model to enrich the existing data.
One possible conclusion is that throwing a raw dataset into AutoML produces decent and usable results. However, if you want perfect results, you have to understand your data and engineer it upfront. While I did the engineering with pure Python code, one could use no-code solutions to achieve the same. Consider Microsoft Power BI or even Excel 😀.
If you want to learn how to do EDA and data cleaning in Power BI, just read my other stories:👉 68–95–99.7 — The Three-Sigma Rule of Thumb Used in Power BI👉 Are you living in a happy country? Let’s find out with Power BI
What are your thoughts on hand-made vs. AutoML models?
👉, — Sebastian