poldrack/regressioncv — pull request #1 by yarikoptic (2012-12-17 17:18:49+00:00)
Title: _rand.py: compatible with released sklearn 0.12.1 + estimate means of corrcoef across splits

Since corrcoef involves standartization of the data, mean of corrcoef across splits would be different from corrcoef estimated on the whole series of target, predicted values. Since mean and variance of the estimated time series could vary across splits, if we take corrcoef of the whole target,predicted series, the estimate might get driven primarily by those instead of actual correlation (e.g. in my EEG/fMRI transfusion I saw high correlation coef with prediction, although predicted variance was way off the original one).

So -- why not to rely on the mean of estimates across splits? This pull request adds estimation of those for KFold case (obviously corrcoef is NA for LOO). Here are the scatter plots against old results + histograms:

http://nbviewer.ipython.org/url/www.onerussian.com/tmp/regression_cv_demos_rand.ipynb

As you see -- results seems to become (more) symmetric around 0 although I am not yet sure why they become so strongly bimodal ;)