scikit-learn/scikit-learn — pull request #975 by emanuele (2012-07-23 15:34:05+00:00)
Title: GBClassifier, MultinomialDeviance: added logsumexp and nan_to_num to avoid underflows and NaNs

Following my recent (20120723) email on the mailing list, here is the PR to address the issue of numerical underflows and NaNs when using GradientBoostingClassifier and MultinomialDeviance with a large number of classes and a large dataset. The issue boils down to:
1) Computing log(exp(array).sum()) which generates (well known) underflows when the array has values of different order of magnitude. The solution uses sklearn.utils.extmath.logsumexp() as suggested by Gael instead of numpy.logaddexp.reduce().
2) Divisions are protected with nan_to_num to avoid NaNs in the results.

At the moment there is no toy example to test this issue easily. I'll try to work on it. Help is warmly welcome. Anyway this changes solve the problem I observed on my dataset.