Building a product growth model
By: Jan Osolnik
Tags: Growth Hacking, Machine Learning, Startup, Data Science, Fintech

Implementation

The model is meant to be used to change the input spend (especially by the product and marketing teams) and see how that translates to the overall growth of the client base over time. This is used in the signup sub-model to map it to the number of signups. Besides that, certain improvements that are expected in the product are quantified as either increased conversion rate or retention rate (in %, such as 110% meaning 10% improvement in retention because of some new product feature or improved conversion rate because of some decrease of friction in the conversion funnel).

Below we can see an example scenario where we expect an improved conversion rate from February 2019 on, combined with improved retention rate from April 2019 on.

Example input sheet: Future spend and hypothesized product improvement impact

Signup sub-model

The signup sub-model was built using a Lasso linear regression model, mapping the historical spend to CPS (cost per signup). This was done as the relationship is mostly linear — increasing marginal cost per signup as the spend increases.

The features are the country in which the money was spent in, whether a particular month was a crypto craze month (to capture outliers using a boolean feature), and the month of spend (to capture seasonal effects, using a label binarizer). The crypto craze monthly cohorts are treated differently as we had a large inflow of new clients in that period that are not representative of the whole client base, therefore potentially introducing noise in the training data. This was one of the hypotheses that we tested in the feature engineering process and that proved to have enough feature importance to be used.

Lasso regularization was used in order to minimize over-fitting as the number of features increased considerably relative to the size of the data. The alpha parameter was chosen using the grid search with cross-validation as the runtime was not an issue considering the size of the available data. The seasonal effects are the hardest part to model as the amount of spend in a particular month is decided by the performance marketing team leveraging their domain expertise (thus not captured in the data). This way some months have naturally more spend on average than others which makes it harder for the model to generalize. Also, the product is available in certain countries for less time than others which poses additional challenges. We tested Catboost regression model for a different treatment of categorical features but the performance didn’t improve so we decided for a simpler model instead.

With the sub-model trained, we iterated through all the countries and cohort months to forecast the CPS for the whole range of spend that we expected in the future with certain step size, say from 0 euros to 10.000 euro with a step of 1000 euro. The CPS values are then multiplied with the spend itself in order to extract the forecasted signups. This is retrained once a month to add new training values.

Applying the Lasso linear regression (left graph) and extracting the desired relationship (right graph)

Conversion sub-model

To model the relationship with how new users convert over time we used Scipy’s curve fit functionality. We tried different methods and this one provided the best performance considering the complexity.

The modeling is performed on a country level. First, we extract the retention of each of the monthly cohorts of a country. This is followed by building a retention model for a particular country, using all of the existing cohorts as training data.

Then, we iterate through all of the cohorts in that country and have certain conditions based on which a different curve is applied to extrapolate future retention from:

If a cohort has a long enough tenure and is large enough (in tenure 0), we use the curve fit function on that specific cohort retention data and extrapolate into the future

If a curve cannot be fit on the cohort data but it satisfies both conditions mentioned above, we use the country-level curve and apply the peg rate (adjusting the country model with the currently available cohort retention data)

If either of the conditions is not satisfied, we apply the fitted country curve

The function used to model conversion (using Scipy’s curve_fit)

Retention sub-model

The business logic behind the Retention sub-model is mostly re-used from the Conversion sub-model. What is changed is the curve fit function applied to fit the cohort retention data.

The function used to model retention (using Scipy’s curve_fit)

Combined output

The first pipeline starts with the future marketing spend as input into the Signup sub-model from which we extract the number of new signups. This is used as the input into the Conversion sub-model from which we calculate the number of conversions over time (cohort’s tenure) by taking into account the potential improvements in the product’s conversion rate. The output is the new clients in a particular month.

After that, the second pipeline starts by using the input of the number of clients in the current month (period t) and forecasts the number of returned clients in the next month with an expanding window method (period t + 1), across all countries and all monthly cohorts. This is done by taking into account historical retention data and potential future improvements in the product’s retention rate. The output is the number of returned clients in a particular month.

The output of both pipelines is combined into the number of active clients in a particular month (MAU, or in this case monthly active clients).

Evaluation

For the last 6 months that the model was used, the average error in the forecasted number of new clients was 11.1%. The average error in the number of returned clients was 1.7%. This was all evaluated on forecasting t+1, a month ahead. The performance bottleneck was mostly the Signup sub-model which makes sense considering the data that we’re trying to fit and all the exogenous variables that are not included in the model (performance marketing trends over time, future market volatility, market penetration/saturation over time, etc.)

What we also noticed is that if we modeled MAU further in the future by simulating as if the model was implemented earlier, the errors accumulate quickly. Even if percentage-wise the model performs well, the model is not useful for the decision-making process in a year or more in advance as small errors accumulate over a year miss the realized value too much for it to be useful. Whereas there is no quantification of uncertainty embedded into the model (via stochastic processes), we can build some understanding of the reliability and consistency of results with this approach.

Taking this into consideration, we can go more in-depth to have insight into where there are discrepancies between the forecasted and the realized values when forecasting for period t+1 (next month). Is the CPS (cost per signup) in a particular country higher or lower than expected? Have more or fewer users converted? Did we retain more or fewer clients while taking into account the number of forecasted signups? It’s also possible to go deeper and see in which country and which monthly cohort a different number of clients were retained more or less than expected. This can give us insight where certain cohorts in certain countries could be churning more than expected by which we can then go deeper and see some behavioral characteristics that bind them. These are some of the potential use-cases that his model can be used for.

Conclusion

One of the challenges, particularly in the Signup sub-model is that while the data can be relatively large at the start, we need to get to the granularity that is used for the decision-making process to make it insightful (or at least, as close as possible). When split by country there is a lower signal to noise ratio for each of the countries which makes it harder to model. It would be ideal to model down to the acquisition channel but the data is just not large enough to generalize from that level.

The biggest challenge was thinking through the problem formulation and the process of building the data pipeline (and with that, how to modularize the code in a way to make it as easy as possible to maintain and improve over time). Much less time was invested in the process of algorithm selection and tuning.

An obvious shortcoming of the model is that there is no estimation of the uncertainty in the forecasts. As mentioned, we used the expanding window approach to derive a general understanding of the reliability and performance of the output. Additionally, there are potential Black swan events that could occur in the financial markets (in either direction) which would considerably influence the emergent behavior of the client base. By definition, these are high impact events but also practically impossible to forecast so we don’t model them either.

Overall, the model proves to be useful to measure how certain changes in the product’s funnel are reflected in the growth of the client base over time. Besides that, quantifying the effect of the diminishing returns of the user acquisition spend is useful to better understand the business levers of growth which can support the decision-making process.

Sources: