Machine Learning Introduction 2: Our first Example | by Lee Tanenbaum | Towards Data Science

This post follows Machine Learning Introduction 1. We will apply machine learning to marketing analysis as a simple use case with business application. We will survey popular machine learning approaches applying each to marketing analysis. We observe some approaches are more interpretable as measuring return on investment (ROI) of different marketing actions, and other approaches provide more accurate sales forecasts. See the first lecture for an explanation of each of the approaches used in this post.
This content is meant to be the second half of the first lecture on deep learning.
This post gives a real-world example of regression, feature engineering, and using a neural network to model a dataset. It motivates and showcases each method.
In this problem, we are a fictional marketing organization. We look at our historical spend and sales, and map daily marketing spend to daily sales.
Our environment is simplified to be suitable as a learning tool for introductory machine learning. In this setting, we have many days of brand sales, but every day the sales of the brand are only dependent on how much the company spends on that day.
In this example, there are two types of marketing spend, brand marketing (brand) to create an image of the brand, and direct to consumer marketing (d2c) to drive sales.
We simulate this data based on the rule that both types of marketing add to sales, but they contribute most if there is a balance between the two. The graph of spend vs sales is as follows:
Individual results could be visualized like an excel table such as:
We start by importing some packages we will need
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom mpl_toolkits.mplot3d import Axes3D
We set a random seed so that if you perform this on your local machine you will see the same random results as we see here. This is important for reproducibility.
np.random.seed(0)
Next we generate the dataset
NUM_SAMPLES = int(2e3)brand_spend = np.random.rand(NUM_SAMPLES)d2c_spend = np.random.rand(NUM_SAMPLES)individual_contributions = brand_spend * .01 + d2c_spend * .02collaboration = np.square(brand_spend) * d2c_spend * 10sales = individual_contributions + collaboration + np.random.rand(NUM_SAMPLES)/10
Finally for the linear regression section, we have a function to perform linear regression and another function to graph the results:
Here we wrap the dataset into a DataFrame. This lets us view the dataset like an excel table within python.
dataset = pd.DataFrame({    'brand': brand_spend,    'd2c': d2c_spend,    'sales': sales}).round(5)
Which can be viewed by
dataset.head()
Next, we split our dataset into X and Y, and visualize the values.
X = dataset.drop('sales', 1)Y = dataset['sales']X1 = X['brand']X2 = X['d2c']fig = plt.figure(dpi=80, figsize = (10, 4))ax = fig.add_subplot(111, projection='3d')ax.scatter(X1, X2, Y, c='r', label='real sales', s = 1)ax.set_xlabel('Brand Marketing')ax.set_ylabel('Direct to Consumer')ax.set_zlabel('Sales')plt.legend()plt.savefig('images/data.jpg')
Concept Check: Build a dataset yourselves in Python:Build and visualize a similar dataset with linear and custom mathematical relations between marketing activities and sales.
Can you see the interaction you built between the marketing actions?
Try to parameterize your system in the beginning of your script to make it easy to enable or disable interactions.
This is one alternative dataset with slightly different features.
import numpy as npimport pandas as pdimport matplotlib.pyplot as pltfrom mpl_toolkits.mplot3d import Axes3Dnp.random.seed(0)NUM_SAMPLES = int(2e3)noise_ratio= 1e-2brand_spend = np.random.rand(NUM_SAMPLES)d2c_spend = np.random.rand(NUM_SAMPLES)geo_mean_mult = 10exp_brand_spend_mult = 1contributions =  {'brand':.01, 'd2c':.02}brand_contributions = brand_spend * contributions['brand']d2c_contributions = d2c_spend * contributions['d2c']geo_contribution = np.sqrt(brand_spend* d2c_spend) * geo_mean_multexp_brand_contribution = np.exp(brand_spend) * exp_brand_spend_multsales = brand_contributions + d2c_contributions + exp_brand_spend_mult + np.random.rand(NUM_SAMPLES) * noise_ratiodataset = pd.DataFrame({    'brand': brand_spend,    'd2c': d2c_spend,    'sales': sales}).round(5)X = dataset.drop('sales', 1)Y = dataset['sales']X1 = X['brand']X2 = X['d2c']fig = plt.figure(dpi=80, figsize = (10, 4))ax = fig.add_subplot(111, projection='3d')ax.scatter(X1, X2, Y, c='r', label='real sales', s = 1)ax.set_xlabel('Brand Marketing')ax.set_ylabel('Direct to Consumer')ax.set_zlabel('Sales')plt.legend()plt.savefig('demo.jpg')
To learn about the impact of marketing spend, we will start by using linear regression to map marketing spend to sales. We will then look at our loss curves and recognize the deficiency of linear regression, encouraging us to perform feature engineering.
After manual feature engineering, to help with the feature generation process we use a hidden layer of a neural network to extract features, and use those for our final regression problem. We will compare and contrast interpretabilities and performance.
To learn the relation between sales and marketing spend, we start by fitting a linear regression model.
This is to tell the model to fill in the gaps in the following formula:
sales = __ * brand_marketing + __ * direct to consumer + __
Our linear regression approximated this curve by learning the parameters:
sales = 5.11 * brand_marketing + 3.28 * direct to consumer — 2.45
This would be very insightful because we can directly read the ROI of each type of spend, for instance we could read this as measuring brand marketing to have ROI 5, and d2c to have ROI 3. Except we note a problem! We can visually see below that the model does not accurately represent the data. The predictions the model has learned are visualized below in blue, clearly not fitting the sales in red:
The following is some simple code for training a linear regression model and visualizing predictions.
from sklearn.neural_network import MLPRegressorfrom sklearn.linear_model import Ridgefrom mpl_toolkits.mplot3d import Axes3Ddef model_sales_regression(dataset, model='Ridge'):    num_samples = dataset.shape[0]    cutoff = (num_samples * 3) // 4    Xtrn = dataset.drop('sales', 1).iloc[:cutoff,:]    Ytrn = dataset['sales'].iloc[:cutoff]    Xval = dataset.drop('sales', 1).iloc[cutoff:,:]    Yval = dataset['sales'].iloc[cutoff:]    model = Ridge().fit(Xtrn, Ytrn)    coefs = model.coef_.round(2)    yhat = model.predict(dataset.drop('sales', 1))    yhatval = model.predict(Xval)    loss = np.square(Yval - yhatval).mean()        print('learned coefficients', list(zip(X.columns, coefs)))    print('loss:', loss)    print('intercept', model.intercept_.round(2))        return model, yhat, coefs, lossdef graph_real_and_predicted(dataset, yhat, fname=None):    X = dataset.drop('sales', 1)    Y = dataset['sales']    X1 = X['brand']    X2 = X['d2c']    fig = plt.figure(dpi=80, figsize=(10, 4))    ax = fig.add_subplot(111, projection='3d')    ax.scatter(X1, X2, Y, c='r', label='real sales', s = 1)    ax.scatter(X1, X2, yhat, c='b', label='estimated sales', s = 1)    ax.set_xlabel('Brand Marketing')    ax.set_ylabel('Direct to Consumer')    ax.set_zlabel('Sales')    plt.legend()    if fname is not None:        plt.savefig('images/' + fname + '.jpg')
This is the code to train linear regression.
model, yhat, coefs, loss = model_sales_regression(dataset)graph_real_and_predicted(dataset, yhat, 'linear')
To summarize the accuracy of our model, we look at the squared error between the true sales and our predictions. We take the mean average of these squared errors to represent our performance.
This is a common default choice based on minimizing worst-case scenarios. Let’s say an error of 1 is acceptable, and if we frequently error in prediction by 1 that’s not perfect, but we can handle it. But if even one time we’re off by 3, that’s unacceptable. We never want to be off by a large amount, but we’re okay with often being off by small amounts. Well the squared error would square 1 to 1, which is fine, and it would square 3 to 9 which is a much larger error.
Squared error is a common error metric because it minimizes worst-case error (or let’s think of it that way and table the other mathematical reasons for later).
Discuss in groups if reading in a class setting.
1- mean absolute error (the mean of the absolute error of our predictions) is meaningful, but doesn’t scale up quickly as you make large errors. mean(abs(Y-yhat))2- mean absolute percent error (By what percent are we off on average) is meaningful if we are concerned with our percent error more than our actual error.mean(abs(Y-yhat)/Y)
To ensure we compute a realistic measure of our performance, we measure our loss on data points that the model didn’t train on. So we started with some data, used some of it to fit the parameters of the model, and used the other data points to test our performance. This is close to measuring how well the model would perform on unseen data in the future.
With linear regression, our mean squared error is 0.89. This is not meaningful yet, but will be meaningful later in comparison to other models.
However, we can clearly see that our model is not accurately capturing trends in sales, because we see regions of predictable discrepancies between our predictions and true sales.
As a smart professional, we conclude linear regression from spend isn’t able to capture all the structure of the data. To expand our model, we notice the significant increase when both inputs are large, so maybe we need a feature that captures a combination of the two spend types. Therefore, we generate a feature that is the product of brand marketing * direct to consumer marketing spend. For this feature to be large, it requires both brand and d2c spend to be high.
Let’s quickly examine this new feature, brand * d2c.
What happens when we have 0 dollars to spend? The value is 0. What if brand is 1 and d2c is 1? The value is 1 * 1 = 1. So it measures marketing spend?
But what if we have $10 to spend, and we spend $1 on d2c, and $9 on brand? We have 1 * 9 = 9. Okay, but what if we took that same dollar, and split it between the two types of spend, $5 each. We have 5 * 5 = 25! This feature detects “there is a balance where both spend types have investment”.
Now, our model learns to approximate the curve much better, with the function:
sales = 0.68 * brand — 1.01 * d2c + 8.74 * brand * d2c — 0.29
Before running linear regression and visualizing the predictions, try to build the feature brand * d2c in the pandas DataFrame. In the past this was a very meaningful, thoughtful task that was a large part of ML.
Solution Below:
dataset['brand * d2c'] = dataset['brand'] * dataset['d2c']model, yhat, coefs, loss = model_sales_regression(dataset)graph_real_and_predicted(dataset, yhat, 'feature_engineering')
This looks good. Our predictions are now tracking sales better, but we’re still not perfect. Our error is now 0.2, down from .9 in pure linear regression.
Looking at our parameters, we can see that the combination of brand and d2c marketing is the strongest influencer, with a coefficient of 8.74. This is just an approximation of the true sales, but that parameter can be interpreted as such: if brand spend is 0, then increasing d2c spend will do nothing, because brand * d2c will be 0 regardless.
If brand is 0, then ROI from d2c to spend of the combination term is 0.
However, if brand spend is full at 1 (thousand dollars), then as we increase d2c spend we will receive an ROI of 8.74 sales dollars per spend dollar.
If brand is 1, then ROI from d2c to spend of the combination term is 8.74.
This doesn’t sound that reasonable in real dollars, but as an approximation it tells us that the best way to increase sales is to have a balance between brand and d2c marketing, because the combination is the largest factor.
At this time, we recognize that we still have a significant loss of 0.2, and still have visible regions of predictable error, and consider that we can still improve our model.
To add more generated features without having to define what they should be, we consider a small neural network that generates 6 features and then uses those features to predict sales.
Use MLPRegressor from sklearn.neural_network to generate features and model sales with 6 hidden units, then show the features that the model learned.
Below is code that splits up the dataset as before, but uses a Neural Network. It then pieces the coefficients together to report the model representation.
def model_sales_MLP(dataset, hidden, print_coefs = True, max_iter= 10000):    num_samples = dataset.shape[0]    cutoff = (num_samples * 3) // 4    Xtrn = dataset.drop('sales', 1).iloc[:cutoff,:]    Ytrn = dataset['sales'].iloc[:cutoff]    Xval = dataset.drop('sales', 1).iloc[cutoff:,:]    Yval = dataset['sales'].iloc[cutoff:]    model = MLPRegressor(hidden, validation_fraction = 0, solver='lbfgs', max_iter= max_iter).fit(Xtrn, Ytrn)    coefs = model.coefs_    yhat = model.predict(X)    yhatval = model.predict(Xval)    loss = np.square(Yval - yhatval).mean()    hiddens = coefs[0].T    final_mlp = coefs[1].flatten()        coefs = list(zip([dict(zip(X.columns, h)) for h in hiddens],                     [['output mult:', m] for m in  final_mlp.flatten()],                      [['intercept:', i] for i in  model.intercepts_[0]]))    print('loss:', loss)    if print_coefs:        for idx, c in enumerate(coefs):            f1, o, i = c            print('feature', idx, '=', f1['brand'].round(2), '* brand +',                   f1['d2c'].round(2), '* d2c', '+', i[1].round(2))        output = 'yhat = '        for fidx, v in enumerate(final_mlp):            output = output + str(v.round(2)) + ' * feat ' + str(fidx) + ' + '        output = output + str(model.intercepts_[1][0].round(2))        print(output)    return model, yhat, coefs, loss
And we create the model and graph it from our dataset, with a single hidden layer of 6 units
model, yhat, coefs, loss = model_sales_MLP(dataset, [6])graph_real_and_predicted(dataset, yhat, 'neural_network')
After training our neural network, we inspect the features of the model: feature 0 = 2.85 * brand + 1.83 * d2c -2.79feature 1 = 1.62 * brand -2.15 * d2c -0.03feature 2 = 0.08 * brand -0.54 * d2c -0.77feature 3 = 0.55 * brand + 0.16 * d2c -0.81feature 4 = 3.83 * brand + 2.27 * d2c + 0.86feature 5 = 1.6 * brand + 1.78 * d2c + 0.7
The features are then clipped to set any negative values to zero, before being used to predict yhat as follows:
if feature[i] is positive: feat[i] =feature[i]
if feature[i] is negative: feat[i] = 0
Yhat = 3.0 * feat[0] -1.48 * feat[1] + 0.82 * feat[2] -0.2 * feat[3] + 1.6 * feat[4] -1.82 * feat[5]–0.56
Wow is that hard to interpret! In a later lecture, we will get into interpreting neural networks, but we will definitely not be trying to read into each of these individual features, that would be way too tedious.
But look at our loss, we are down to a loss of 0.12! Also, our graph now looks to track very closely with true sales.
Lastly, let’s assume that we are not trying to measure ROI to inform marketing decisions but just trying to forecast daily sales. Rather than 6 hidden units (6 features) in our neural network, we instead use 100 hidden units, and use that hidden layer to compute a new hidden layer of 100 units, that we finally use to perform linear regression.
This reduces our loss much further, and now we are almost perfectly fitting to the dataset at a loss of 0.001. Now think, this isn’t even training data. This isn’t how well the model fits to the data it was trained on, this is how well the model fits to data it had never seen during training.
But what if we wanted to interpret this model? We have a hidden layer of 100 units, and then another hidden layer of 100 units. So to compute the second hidden layer we compute 100 multiplications from the first hidden layer for each of the 100 units in the second hidden layer. That’s 10,000 multiplications it learned! We could never conceptualize that many interactions, but we recognize the neural network has developed these features to accurately fit the data.
So let’s recap. Initial linear regression modeled the dataset, but it clearly had regions it didn’t fit well. We performed some feature engineering because we had a guess of some valuable features from thinking of the problem. This helped us fit the dataset much better, and our model was more accurate but a little harder to understand.
Next we introduced a hidden layer into the network to learn our features, and we came up with six features that somehow solved the problem for us. These became even more difficult to interpret, but we know these features somehow understand the model much better than our original two or three.
We wrapped up by letting a deep neural network solve the problem. We observed that this model is much more accurate, but interpreting the tens of thousands of multiplications would not be feasible.
Below we will re-perform all the analysis above quickly, while only reporting the code and brief summaries. First, we import some libraries that will help us later
import numpy as npimport pandas as pdimport matplotlib%matplotlib notebookimport matplotlib.pyplot as pltfrom sklearn.neural_network import MLPRegressorfrom sklearn.linear_model import Ridgefrom mpl_toolkits.mplot3d import Axes3D
We set a random seed so that if you perform this on your local machine you will see the same random results as we see here. This is important for reproducibility.
np.random.seed(0)
Next we generate the dataset
NUM_SAMPLES = int(2e3)brand_spend = np.random.rand(NUM_SAMPLES)d2c_spend = np.random.rand(NUM_SAMPLES)individual_contributions = brand_spend * .01 + d2c_spend * .02collaboration = np.square(brand_spend) * d2c_spend * 10sales = individual_contributions + collaboration + np.random.rand(NUM_SAMPLES)/10
Finally for the linear regression section, we have a function to perform linear regression and another function to graph the results:
def model_sales_regression(dataset, model='Ridge'):    num_samples = dataset.shape[0]    cutoff = (num_samples * 3) // 4    Xtrn = dataset.drop('sales', 1).iloc[:cutoff,:]    Ytrn = dataset['sales'].iloc[:cutoff]    Xval = dataset.drop('sales', 1).iloc[cutoff:,:]    Yval = dataset['sales'].iloc[cutoff:]    model = Ridge().fit(Xtrn, Ytrn)    coefs = model.coef_.round(2)    yhat = model.predict(dataset.drop('sales', 1))    yhatval = model.predict(Xval)    loss = np.square(Yval - yhatval).mean()        print('learned coefficients', list(zip(X.columns, coefs)))    print('loss:', loss)    print('intercept', model.intercept_.round(2))        return model, yhat, coefs, lossdef graph_real_and_predicted(dataset, yhat, fname=None):    X = dataset.drop('sales', 1)    Y = dataset['sales']    X1 = X['brand']    X2 = X['d2c']    fig = plt.figure(dpi=80, figsize=(10, 4))    ax = fig.add_subplot(111, projection='3d')    ax.scatter(X1, X2, Y, c='r', label='real sales', s = 1)    ax.scatter(X1, X2, yhat, c='b', label='estimated sales', s = 1)    ax.set_xlabel('Brand Marketing')    ax.set_ylabel('Direct to Consumer')    ax.set_zlabel('Sales')    plt.legend()    if fname is not None:        plt.savefig('images/' + fname + '.jpg')
Here we wrap the dataset into a DataFrame. This lets us view the dataset like an excel table within python.
dataset = pd.DataFrame({    'brand': brand_spend,    'd2c': d2c_spend,    'sales': sales}).round(5)
Which can be viewed by
dataset.head()
Next, we split our dataset into X and Y, and visualize the values.
X = dataset.drop('sales', 1)Y = dataset['sales']X1 = X['brand']X2 = X['d2c']fig = plt.figure(dpi=80, figsize = (10, 4))ax = fig.add_subplot(111, projection='3d')ax.scatter(X1, X2, Y, c='r', label='real sales', s = 1)ax.set_xlabel('Brand Marketing')ax.set_ylabel('Direct to Consumer')ax.set_zlabel('Sales')plt.legend()plt.savefig('images/data.jpg')
This is the code to train linear regression.
model, yhat, coefs, loss = model_sales_regression(dataset)graph_real_and_predicted(dataset, yhat, 'linear')
Next, we add a feature and perform linear regression from the augmented dataset.
dataset['brand * d2c'] = dataset['brand'] * dataset['d2c']model, yhat, coefs, loss = model_sales_regression(dataset)graph_real_and_predicted(dataset, yhat, 'feature_engineering')
Moving to neural networks, we define our Multi layer perceptron (another term for neural network) function.
def model_sales_MLP(dataset, hidden, print_coefs = True, max_iter= 10000):    num_samples = dataset.shape[0]    cutoff = (num_samples * 3) // 4    Xtrn = dataset.drop('sales', 1).iloc[:cutoff,:]    Ytrn = dataset['sales'].iloc[:cutoff]    Xval = dataset.drop('sales', 1).iloc[cutoff:,:]    Yval = dataset['sales'].iloc[cutoff:]    model = MLPRegressor(hidden, validation_fraction = 0, solver='lbfgs', max_iter= max_iter).fit(Xtrn, Ytrn)    coefs = model.coefs_    yhat = model.predict(X)    yhatval = model.predict(Xval)    loss = np.square(Yval - yhatval).mean()    hiddens = coefs[0].T    final_mlp = coefs[1].flatten()        coefs = list(zip([dict(zip(X.columns, h)) for h in hiddens],                     [['output mult:', m] for m in  final_mlp.flatten()],                      [['intercept:', i] for i in  model.intercepts_[0]]))    print('loss:', loss)    if print_coefs:        for idx, c in enumerate(coefs):            f1, o, i = c            print('feature', idx, '=', f1['brand'].round(2), '* brand +',                   f1['d2c'].round(2), '* d2c', '+', i[1].round(2))        output = 'yhat = '        for fidx, v in enumerate(final_mlp):            output = output + str(v.round(2)) + ' * feat ' + str(fidx) + ' + '        output = output + str(model.intercepts_[1][0].round(2))        print(output)    return model, yhat, coefs, loss
We use our function with the raw sales and spend data, without the augmentation features.
dataset = pd.DataFrame({    'brand': brand_spend,    'd2c': d2c_spend,    'sales': sales}).round(5)model, yhat, coefs, loss = model_sales_MLP(dataset, [6])graph_real_and_predicted(dataset, yhat, 'neural_network')
Finally, we call the function again, but with a deeper neural network.
model, yhat, coefs, loss = model_sales_MLP(dataset, [100, 100], max_iter = 1000, print_coefs=False)graph_real_and_predicted(dataset, yhat, fname = None)
And that’s an introduction to machine learning from the practical point of view of marketing attribution and estimation!
The code for the above is: here
This problem expands upon: ML Preface.
Intended Follow up within this series: https://towardsdatascience.com/ml-intro-3-logistic-output-units-ec42cc576634
Intended Follow-Up Post for those who fully understand this one:https://medium.com/@leetandata/machine-learning-engineering-1-custom-loss-function-for-house-sales-estimation-95eec6b12457
Optional Follow-Ups for those struggling with code or understanding:Detailed analysis of the code in this posting: https://medium.com/@leetandata/machine-learning-python-programming-introduction-for-business-people-10588e13ce9d
Free comprehensive class on Python Programming. Make sure to enroll but choose the free optional. All class content is freely available:https://www.coursera.org/learn/python-programming-introduction/
A Machine Learning tutorial more complicated than this one, but easier than the following (that I didn’t write) https://www.kaggle.com/rochellesilva/simple-tutorial-for-beginners
Optional Follow-Ups Posts for Software Engineers:Math- and CS- heavy detailed explanation (Part 1): https://medium.com/@leetandata/neural-network-introduction-for-software-engineers-1611d382c6aa
Math- and CS- heavy detailed explanation (Part 2): https://medium.com/@leetandata/neural-network-for-software-engineers-2-mini-batch-training-and-validation-46ee0a1269a0
Optional Math Notes from Professor Cho:https://github.com/nyu-dl/Intro_to_ML_Lecture_Note/raw/master/lecture_note.pdf