mne-tools/mne-python — pull request #707 by mluessi (2013-07-31 20:42:21+00:00)
Title: Use joblib memmapping pool (finally :) )

Joblib now supports sharing memory using memmaping (see joblib/joblib#44), which can result in large memory savings when using parallel processing with large read-only memory blocks.

This PR is an update of #99 and will solve #95. To use this feature, the user has to use `mne.set_cache_dir`. Also, the feature is currently only used in the cluster-stats code, where I made some modifications to use block-wise processing to save memory.

Usage example:

import numpy as np

from mne.stats import permutation_cluster_1samp_test

n_subjects = 100
n_vars = int(1e6)

rng = np.random.RandomState(42)
X = rng.randn(n_subjects, n_vars)
X[:, 400:1000] += 100

print 'data size: %f MB' % ((8. * X.size) / 2 ** 20)

import mne
mne.set_cache_dir('/dev/shm')
#mne.set_cache_dir(None)

T_obs, clusters, p_values, H0 = \
permutation_cluster_1samp_test(X, n_jobs=8, threshold=100.0,
tail=1, n_permutations=32)

On my workstation, without memmaping, this needs ~7GB of RAM, while with memmaping it needs ~1GB.

Note: You need the latest version of joblib from github to benefit from this.