I agree with you on that point but then the problem is that it can take a significant amount of memory (its size is n_samples * n_perm).