opened
bdwilliamson
username_0
@username_1:

When the number of elements in a class for a dichotomous outcome is <= the number of folds, we'll run into lots of numerical instability: we're running stratified CV, which means that each fold will have <= 1 of the small class. This leads to even worse problems for the CV-superlearner, in which case we may not have any of the small class. It also led to a fatal error message when computing variable importance since I split the sample in two prior to doing importance computation.

My gut reaction is to print an error message and don't run the analysis for any outcome with <= V in one class. Also, for any outcome with <= V in one class after sample-splitting, print an error message and don't run variable importance for that outcome. Thoughts?
Errors when number of elements in class is <= number of folds (dichotomous outcome)
issue
created
benkeser
username_1
Agreed. I would throw an error early on.
comment
created
bdwilliamson
username_0
Fixed in #17
comment
closed
bdwilliamson
username_0
issue