Repository: scikit-learn/scikit-learn
Pull request: DOC Refer to ONNX
Arguably fixes #11041

Problem statement:
Best practices for deploying scikit-learn models?
What are the best practices for deploying scikit-learn models on the cloud? I have trained a Random Forest Classifier. Models size is approx 2gb and gunicorn server is crashing on production. How to use huge ML models in Flask App with gunicorn server

Hints from issue discussion:
I think this is a good question for us to document answers to, for example
in the FAQ. one option: train and pickle within a docker and provide a web
service frontend for prediction (perhaps there are existing
implementations). Another: export a pmml representation and use pmml
infrastructure. But these are not answers from experience and it would be
good to hear from others. Perhaps ask on our mailing list.

https://github.com/onnx/onnx might become an option as well.
can you please little bit explain how [https://github.com/onnx/onnx](url) can be helpful? @amueller 
@vimox-shah This is exactly how we do it. GCS has the Kubernetes engine handles container orchestration so you have all your models (and their containers) in one place. Can't speak for AWS though. Not sure what cloud platform you're using.
> one option: train and pickle within a docker and provide a web
service frontend for prediction

Additionally, in some cases we use Flask and provide an API for consumers of the model results and in other cases, simple cron jobs are used to run a batch script that scores data and pushes it to our CRM tool via the API the CRM tool provides. But containers are the best way (imo) to deploy your models and using a container orchestration platform like Kubernetes allows you to scale automatically as needed.
sorry @jmwoloso I didn't get what you want to say. Can you please explain in different way?
@vimox-shah I don't think I can make it much simpler, but...:

1. Train model
2. Serialize model (using `joblib from sklearn.externals`)
3. Deploy model using containers (Docker)
4. Use Kubernetes to manage containers and resources


Inside those containers, you can have your `flask` apps. In flask you build endpoints that can be either internally used by your script and other functions, or accessible to the outside world via webpages or a API that you specify. 

I was simply adding a little more specificity/context to the first answer proposed to you by @jnothman. Hope that helps.
what does 3 point Deploy model using containers (Docker) mean? Do I need to create a image of that model and push to docker hub and use that image or should I use persistent storage in kubernetes for model file? @jmwoloso 
Can I use google cloud storage where I can store model and install in the container when my flask app goes live. model will load only once when the server starts. So whenever a request comes to the server it uses loaded model. what do you think about this approach? @jmwoloso @amueller 
@vimox-shah Essentially, your flask app lives in the container and where you store your serialized model is up to you depending on whether you're using CI/CD or just updating it every so often.
> what does 3 point Deploy model using containers (Docker) mean? Do I need to create a image of that model and push to docker hub and use that image or should I use persistent storage in kubernetes for model file?

Yes, this should work fine. If you install the command line tool called `gcsfuse` you can mount your GCS buckets directly onto your GCP instances which makes loading the serialized model trivial.
> Can I use google cloud storage where I can store model and install in the container when my flask app goes live. model will load only once when the server starts. So whenever a request comes to the server it uses loaded model. what do you think about this approach? @jmwoloso @amueller

References:
[GCSFuse Installation](https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/installing.md)
[GCSFuse Docs](https://cloud.google.com/storage/docs/gcs-fuse)
Can I take this issue up and add it as new section in the documentation?

Commit message: DOC Refer to ONNX

Patch to review:
diff --git a/doc/faq.rst b/doc/faq.rst
index c49f3df86027e..58c76b2eefa20 100644
--- a/doc/faq.rst
+++ b/doc/faq.rst
@@ -66,6 +66,11 @@ might be found.
 **Please do not email any authors directly to ask for assistance, report bugs,
 or for any other issue related to scikit-learn.**
 
+How should I save, export or deploy estimators for production?
+--------------------------------------------------------------
+
+See :ref:`model_persistence`.
+
 How can I create a bunch object?
 ------------------------------------------------
 
diff --git a/doc/modules/model_persistence.rst b/doc/modules/model_persistence.rst
index ffd345413ce91..90fd51b4c0965 100644
--- a/doc/modules/model_persistence.rst
+++ b/doc/modules/model_persistence.rst
@@ -9,6 +9,11 @@ the model for future use without having to retrain. The following section gives
 you an example of how to persist a model with pickle. We'll also review a few
 security and maintainability issues when working with pickle serialization.
 
+An alternative to pickling is to export the model to another format using one
+of the model export tools listed under :ref:`related_projects`. Unlike
+pickling, once exported you cannot recover the full Scikit-learn estimator
+object, but you can deploy the model for prediction, usually by using tools
+supporting open model interchange formats such as `ONNX`_ or `PMML`_.
 
 Persistence example
 -------------------
diff --git a/doc/related_projects.rst b/doc/related_projects.rst
index 1c2362a2782c9..ad080bcc209cb 100644
--- a/doc/related_projects.rst
+++ b/doc/related_projects.rst
@@ -90,6 +90,10 @@ enhance the functionality of scikit-learn's estimators.
 
 **Model export for production**
 
+- `onnxmltools <https://github.com/onnx/onnxmltools>`_ Serializes many
+  Scikit-learn pipelines to `ONNX <https://onnx.ai/>`_ for interchange and
+  prediction.
+
 - `sklearn-pmml <https://github.com/alex-pirozhenko/sklearn-pmml>`_
   Serialization of (some) scikit-learn estimators into PMML.
 


Code review comments:
diff_hunk: @@ -90,6 +90,10 @@ enhance the functionality of scikit-learn's estimators.
 
 **Model export for production**
 
+- `onnxmltools <https://github.com/onnx/onnxmltools>`_ Serializes many
+  Scikit-learn pipelines to `ONNX <https://onnx.ai/>`_ for interchange and
+  prediction.
+
 - `sklearn-pmml <https://github.com/alex-pirozhenko/sklearn-pmml>`_
path: doc/related_projects.rst
text: @user1:
Do you ever remove projects from this list? The `sklearn-pmml` project was last updated in Oct 2015, and by my (slightly biased-) judgement never reached production quality (claims to support four decision tree ensemble model types, but the exported PMML documents are provably invalid/incorrect).

@user2:
Yeah, the author himself did suggest to use sklearn2pmml in https://github.com/alex-pirozhenko/sklearn-pmml/issues/26#issuecomment-240791654 in 2016.

PR welcome to remove it.

@user2:
or at least add a comment that it's no longer maintained.