Predicting Win Probability. Using machine learning on sales... | by Vikram Devatha | Towards Data Science

Have you ever had to determine the probability of winning opportunities in the pipeline? The Marketing team generates leads which are followed up by the Sales team, and in the process each opportunity goes through a lifecycle, from the RFP stage till final closure. As the opportunity transitions from one stage to the next, the point-of-contact assigns a probability of converting the lead into a sale, and thereby winning the opportunity.
But how accurate is this probability? Given a large enough dataset with multiple predictor variables, can we use machine learning to determine the probability of winning an opportunity? We built a proof of concept at a leading asset management company, to showcase the benefits of letting the machine do the talking :)
Nine different data sources were leveraged to extract 14 attributes at the opportunity level. These included duration of relationship, expected duration of Opportunity, number of CRM activities, opportunity amount, type of opportunity (new, cross sell, etc.), duration since last contact, number of presentations made, name of the presenter, number of product webinars attended, assets managed, and growth in value of assets.
These were transformed into a single data frame with each record representing an opportunity, as follows
The opportunity data frame was then segregated into open and closed opportunities. The final status is known for the opportunities that have closed — either won, lost or inactive, while the status is to be predicted for open opportunities. The status label is transformed into 0 or 1, with 1 representing winning the opportunity, and 0 representing losing the opportunity (including the ‘inactive’ status).
The closed opportunities are split into Training and Test in the 75:25 ratio, while maintaining the ratio of the status label using the createDataPartition function in the Caret package.
> set.seed(101) > intrain = createDataPartition(y=closed_opps$Label, p=0.75, list=FALSE)> traindf = closed_opps[intrain,]; traindf$Label = as.factor(traindf$Label)> testdf = closed_opps[-intrain,]; testdf$Label = as.factor(testdf$Label)
I used the Caret, ARM and XGBoost packages in R to build Supervised machine learning models on the Training data, and classify the Opportunity as either Won or Lost.
Naive Bayes: a classification algorithm based on Bayes Theorem, that determines the probability of winning or losing an Opportunity given that each predictor variable has taken on a certain value.
nb_model <- train(Label ~ ., data = traindf, method = “naive_bayes”, trControl = fitControl, na.action = na.pass)
Logistic Regression: A generalized linear model but with a binary outcome, which is transformed into a probability using the sigmoid function. The ‘weight’ for each predictor variable is determined by the model in order to reduce the error between the actual and predicted values.
lr_model <- train(Label ~ ., data = traindf, method = “glm”, family = “binomial”, trControl = fitControl, na.action = na.pass)
Bayesian GLM: An extension of the logistic regression model, but one that assumes a prior distribution of the predictor variables in accordance with Bayesian methods.
bayenesian_model <- train(Label ~ ., data = traindf, method = “bayesglm”, trControl = fitControl, na.action = na.pass)
Boosted Logistic Regression: an ensemble of several logistic regression models, which are then used to make a prediction based on the individual predictions.
boosted_lr_model <- train(Label ~ ., data = traindf, method = “LogitBoost”, trControl = fitControl, na.action = na.pass)
Random Forest: An ensemble of Decision Trees, used as slow learners that classifies each Opportunity, by bifurcates the training data into the respective Status using the Feature that provides the best split at each node.
library(randomForest) # to build a random forest modelrf_model = train(Label ~ ., data = traindf, method = “rf”, ntree = 30, maxdepth = 5, trControl = fitControl, na.action = na.pass)detach(“package:randomForest”, unload=TRUE) #conflicts with margin in ggplot
Extreme Gradient Boost: Builds an ensemble of Decision Trees in a sequential manner, where the residuals of each model are fit in the subsequent model.
xgb_model <-train(Label ~., data = traindf, method = “xgbTree”, trControl = fitControl, tuneGrid = xgb.grid, verbose = T, nthread = 2, na.action = na.pass)
Extreme Gradient Boost (with DART): An extension of the Extreme Gradient Boost algorithm, but trees are dropped at each stage in order to avoid the problem of over-fitting.
xgbDART_model <- train(Label ~ ., data = traindf, method = “xgbDART”, trControl = fitControl, na.action = na.pass)
Each ML model was then deployed on the Test dataset
nb_predictions_test = predict(nb_model, newdata = testdf, type = “raw”)bayenesian_predictions_test = predict(bayenesian_model, newdata = testdf, type = “raw”)boosted_lr_predictions_test = predict(boosted_lr_model, newdata = testdf, type = “raw”)lr_predictions_test = predict(lr_model, newdata = testdf, type = “raw”)rf_predictions_test = predict(rf_model, newdata = testdf, type = “raw”)xgbDART_predictions_test = predict(xgbDART_model, newdata = testdf, type = “raw”)xgb_predictions_test = predict(xgb_model, newdata = testdf, type = “raw”)
The performance is measured using 3 metrics: ROC, Sensitivity and Specificity. ROC is the Receiver Operating Characteristic, i.e. area under the curve, overall accuracy of the classifier. Sensitivity is the true positive rate, i.e. number of opportunities correctly predicted as wins as a proportion of total wins. Specificity is the true negative rate, i.e. number of opportunities correctly predicted as lost as a proportion of the total number of lost opportunities.
The Random Forest model provided the highest Sensitivity on the Test dataset. This was used to predict the probability of winning the Open Opportunities, and the results were showcased in the following manner.
The algorithm was also used to determine the importance of each input Feature using the varImp function from the Caret package. The results showed that certain features which we assumed would contribute to the model were actually not helping, and this warranted a discussion on whether the data was faulty, or if we had to revise our assumptions on the feature.
Being able to predict the win rate of opportunities in the pipeline will enable us to implement an “early warning system” and also provide a framework for engaging with prospective clients. For instance, CRM activities could be prioritized based on the win-probability as well as opportunity amount. The output of this model could also become the input for a subsequent segmentation model that buckets clients as being strategic or otherwise.
The main challenge we face(d) while working on these models is the lack of clean data. We found several inconsistencies in the way in which data was being maintained in different spreadsheet files, and had to manually map them to the kID in the central repository. We also found that ML models require a lot more data than was available at the time of building the models. As we transition from legacy systems, and clean up the data in parallel, we believe this proof of concept will see the light of day.
If you have any comments or feedback, or another way of predicting the win rate of Opportunities in the pipeline, pl use the comments section below.
Here is a link on the Caret package that you will find useful: