Package: simpleprophet
simpleprophet KPI forecasting models and toolsThis directory contains the simpleprophet forecast models and accompanying tools.  The simpleprophet models prioritize simplicity and stability, adding complexity and "bendiness" only when it significantly improves model performance on a holdback period.SetupYou will need a python environment withfbprophetand a few other
dependencies installed. We provide a Docker image that can be pulled from GCR
and run interactively like:GOOGLE_APPLICATION_CREDENTIALS=/path/to/creds.json
dockerrun-it\-eGOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/key.json\-v$GOOGLE_APPLICATION_CREDENTIALS:/tmp/keys/key.json:ro\--entrypointpython\gcr.io/moz-fx-data-forecasting/simpleprophetOr you can make code updates and build the image locally:dockerbuild.--tagsimpleprophet
dockerrun-it\-eGOOGLE_APPLICATION_CREDENTIALS=/tmp/keys/key.json\-v$GOOGLE_APPLICATION_CREDENTIALS:/tmp/keys/key.json:ro\--entrypointpython\simpleprophetIf you want to produce a local environment outside docker, you can create an
appropriate Conda environment viaconda env create -f environment.yml.
To create an environment outside Conda, see thefbprophetinstallation instructionsand then usepip install -r requirements.txtto install remaining dependencies.UsageThe functions for running the forecasting pipeline are inpipeline.py.  Theupdate_tablefunction can be run daily to add rows to the output table as necessary to incorporate newly available metrics data.  Thereplace_tablefunction will clear the output table and regenerate it from scratch.For model-building the code inmodeling.pymay be useful.  It includes a function to evaluate a model on a holdout set and provide some useful visualizations.Thevalidations.pyfile contains code that produces plots to evaluate model performance and validate the model behavior over time is reasonable.  It can also be used to compare multiple models.Modeling StrategyThemodels.pyfile contains the production model specifications.  The models were developed by Jesse McCrosky using the fbprophet framework.  The guiding modeling philosophy was to be guided by simplicity and intuitive fit, while informing the modeling process using conventional types of quantitative evidence.The evaluation of a forecast is fundamentally multi-dimensional.  As well as the competing objectives of stability, accuracy, and non-bias, each of these objectives can be evaluated on multiple time horizons.  This complexity makes a pure machine learning optimization approach extremely complex.As an alternative, I chose to fit the models somewhat intuitively.  Parameter sets were explored iteratively and each iteration was evaluated visually to see if the model components (seasonality and trend) seemed to fit the observed actuals.  Once a reasonable parameter space was defined, the modeling process proceeded to evaluate holdout set metrics and other quantitative evaluations on a set of models.  Simpler models were preferred and complexity was only added when clearly justified by the quantitative evidence.A few relevant model characteristics:Due to the "smoothed" nature of MAU as a metric, yearly seasonality was adequate to capture all holiday effects except for Easter, which was included as a model component.We select a start date for the training data based on the point where the metric appears to have reached a somewhat steady state in its development - the first weeks of most metrics are quite atypically and their use for training would not be helpful.Similarly, some product metrics have "anomalies" - periods during which the metric value was highly atypical, usually due to a data problem.  These periods were excluded from training data.The appropriate start dates and anomaly periods were determined through manual examination of metric plots.DeployingTo deploy to PyPI, admins on the repo can make a new release following semver
(0.1.0, for example), which will trigger a CircleCI build that publishes
a new package to PyPI with that tag name as the version.For more information, contactjmccrosky@mozilla.com