Repository: scikit-learn/scikit-learn
Pull request: [MRG+1] Add note about the size of a random forest model #6276
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
Fixes #6276 

#### What does this implement/fix? Explain your changes.
Add a note in the section random forest for informing how to reduce the size of the resulting model.

#### Any other comments?
First contribution, remarks are welcome.

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
Need a condensed representation for storing random forest classifiers
scikit-learn version: 0.16.1
python: 3.4.4 (anaconda 2.3.0)
machine: Linux RH 6, more memory than you can shake a stick at

There should be an option to store random forest classifiers that can be used for predictions, in a condensed representation, e.g. with each decision point having the descriptor, operator, and cutoff value, and the leaves having the classification.  I trained a 300 tree model using default parameters, with about 100K data points (about 20 descriptors each).  This trained fine but the resulting model, output with compression by joblib, is 182 MB (!).  I tried training another model that had ten times the data points (same number of trees).  This finished building, but joblib choked with the dreaded "OverflowError: Size does not fit in an unsigned int" error (supposedly this was fixed with python 3?).  

By the looks of it, scikit-learn is storing the data used to build the model as part of the model, with no options not to do this.  Any way around this?  I want to build a model with 4M data points...

Hints from issue discussion:
I have tried all kinds of variants, using gzip and pickle, e.g.:

def saveModel(self, fileName):
        assert self.classifier is not None
        try:
            with gzip.open(fileName, "wb", 3) as gz:
                pickle.dump(self.classifier, gz, protocol=pickle.HIGHEST_PROTOCOL)
            return True
        except IOError as err:
            print("ERROR: Dump of classifier model failed: {}".format(err))
        return False

This gives the same error...  Aargh...

I really don't think the training data is being stored with the model. But
you say you build with default parameters. Have you considered that the
default parameters -- with min_samples_split=2, min_samples_leaf=1,
max_leaf_nodes=None, max_depth=None -- might tend towards large, overfit
models if the data are not easily separable? Could you try joblib.dumping
after limiting the depth?

On 4 February 2016 at 14:04, dgehlhaar notifications@github.com wrote:

> I have tried all kinds of variants, using gzip and pickle, e.g.:
> 
> def saveModel(self, fileName):
> assert self.classifier is not None
> try:
> with gzip.open(fileName, "wb", 3) as gz:
> pickle.dump(self.classifier, gz, protocol=pickle.HIGHEST_PROTOCOL)
> return True
> except IOError as err:
> print("ERROR: Dump of classifier model failed: {}".format(err))
> return False
> 
> This gives the same error... Aargh...
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/scikit-learn/scikit-learn/issues/6276#issuecomment-179592046
> .

Hi,

With default parameters, the size of the forest will be O(M_N_Log(N)),
where M is the number of trees and N is the number of samples. So yes, it
is expected for the model to get large if you build it on an even larger
dataset. As Joel points out, you can usually reduce the size of the model
by setting min_samples_split, max_leaf_nodes or max_depth.

Gilles

On 4 February 2016 at 04:27, Joel Nothman notifications@github.com wrote:

> I really don't think the training data is being stored with the model. But
> you say you build with default parameters. Have you considered that the
> default parameters -- with min_samples_split=2, min_samples_leaf=1,
> max_leaf_nodes=None, max_depth=None -- might tend towards large, overfit
> models if the data are not easily separable? Could you try joblib.dumping
> after limiting the depth?
> 
> On 4 February 2016 at 14:04, dgehlhaar notifications@github.com wrote:
> 
> > I have tried all kinds of variants, using gzip and pickle, e.g.:
> > 
> > def saveModel(self, fileName):
> > assert self.classifier is not None
> > try:
> > with gzip.open(fileName, "wb", 3) as gz:
> > pickle.dump(self.classifier, gz, protocol=pickle.HIGHEST_PROTOCOL)
> > return True
> > except IOError as err:
> > print("ERROR: Dump of classifier model failed: {}".format(err))
> > return False
> > 
> > This gives the same error... Aargh...
> > 
> > —
> > Reply to this email directly or view it on GitHub
> > <
> > https://github.com/scikit-learn/scikit-learn/issues/6276#issuecomment-179592046
> > 
> > .
> 
> —
> Reply to this email directly or view it on GitHub
> https://github.com/scikit-learn/scikit-learn/issues/6276#issuecomment-179597165
> .

Thanks everyone, I will play with parameters and see what comes out.  I tried limiting the depth a bit previously but my error rate went way up.  I will look again.

All, I toyed with parameters, and came up with the following, which gives good performance and "reasonable" file size: 300 trees; min 5 points / leaf; max 22 levels; max 1000 leaves.

Thanks!

This issue many be closed.

> This issue many be closed.

We should add a note in the documentation about this.

GaelVaroquaux, I agree, especially the expected growth in the tree size with default parameters.  I don't have enough experience to know if any of my results can be generalized.  But it is likely that most people will just try with default parameters.  Thanks!

This is indeed a very frequent question.

Is it still up to date ? I can take it, if it is.

Commit message: DOC: Add note about the size of a random forest model (See #6276)

Patch to review:
diff --git a/doc/modules/ensemble.rst b/doc/modules/ensemble.rst
index 52f1f0bc0ff62..bc5326be8d453 100644
--- a/doc/modules/ensemble.rst
+++ b/doc/modules/ensemble.rst
@@ -213,6 +213,13 @@ When using bootstrap sampling the generalization accuracy can be estimated
 on the left out or out-of-bag samples. This can be enabled by
 setting ``oob_score=True``.
 
+.. note::
+
+    The size of the model with the default parameters is :math:`O( M * N * log (N) )`,
+    where :math:`M` is the number of trees and :math:`N` is the number of samples.
+    In order to reduce the size of the model, you can adapt those parameters:
+    ``min_samples_split``, ``min_samples_leaf``, ``max_leaf_nodes`` and ``max_depth``.
+
 Parallelization
 ---------------
 


Code review comments:
diff_hunk: @@ -213,6 +213,13 @@ When using bootstrap sampling the generalization accuracy can be estimated
 on the left out or out-of-bag samples. This can be enabled by
 setting ``oob_score=True``.
 
+.. note::
+
+    The size of the model with the default parameters is :math:`O( M * N * log (N) )`,
+    where :math:`M` is the number of trees and :math:`N` is the number of samples.
+    In order to reduce the size of the model, you can adapt those parameters:
path: doc/modules/ensemble.rst
text: @user1:
I don't think adapt is the right word. You can say "change" or "modify" or "tune"