Repository: nilearn/nilearn

Issue:
Ship compressed versions of the Allen atlases
The Allen atlases that we ship are not compressed. As a result, they use a total of 75M on a download (eg circleCI). We should reship them using OSF, after a "gzip -9", this drops the disk usage to 31Mb.


Discussion:
@kchawla-pi : do you think that you can tackle this in the near future? It should make our circleCI builds a bit faster and a bit more robust, hence streamlining the release.
Yeah. I'll do it tomorrow if I am unavailable today.
Thanks!

What do you think, how far along are we towards the release?
> What do you think, how far along are we towards the release?

Almost there.

Do I unzip the three files in the dataset, then zip them all backup, so there are no nested zips?
Or do I maintain the nested zip file structure?
Oh wait, the files inside are .nii files, they are not compressed at all. Nevermind
The goal is to ship .nii.gz so that less disk space is used.

⁣Sent from my phone. Please forgive typos and briefness.​

On Oct 14, 2018, 19:28, at 19:28, Kshitij Chawla <notifications@github.com> wrote:
>Oh wait, the files inside are .nii files, they are not compressed at
>all. Nevermind
>
>-- 
>You are receiving this because you authored the thread.
>Reply to this email directly or view it on GitHub:
>https://github.com/nilearn/nilearn/issues/1792#issuecomment-429645096

@GaelVaroquaux @bthirion 
I have created a project on OSF to store the data in a manner accessible to the fetchers, on the basis of other osf datasets we are using in the fetcher.
Before I proceed, take a look at it and make sure everything is in the clear, legally & scientific etiquette-wise.
> Before I proceed, take a look at it and make sure everything is in the clear, legally & scientific etiquette-wise.

You should check on the original page that we are allowed to rehost it.
If we are not, then we should consider removing the function from the
examples, so that the atlas is not downloaded.

Will do tomorrow.
Can't seem to find anything about legal rights regarding the data, the paper itself seems open access. Certainly nothing about rehosting the data.
Should I write them an email and clarify?
> Should I write them an email and clarify?

I would just go ahead.

I have uploaded the data as a project and made it public, now figuring out the url.
Not sure why but it keeps glitching, where similar code in our code base from other osf projects is working fine.
Figured it out. Will require more work than I thought, no worries.
@GaelVaroquaux Review & Discussion nescessary for #1840 

Resolving patch:
diff --git a/nilearn/datasets/atlas.py b/nilearn/datasets/atlas.py
--- a/nilearn/datasets/atlas.py
+++ b/nilearn/datasets/atlas.py
@@ -851,17 +851,17 @@ def fetch_atlas_allen_2011(data_dir=None, url=None, resume=True, verbose=1):
     on this dataset.
     """
     if url is None:
-        url = "http://mialab.mrn.org/data/hcp/"
+        url = "https://osf.io/hrcku/download"
 
     dataset_name = "allen_rsn_2011"
     keys = ("maps",
             "rsn28",
             "comps")
 
-    opts = {}
-    files = ["ALL_HC_unthresholded_tmaps.nii",
-             "RSN_HC_unthresholded_tmaps.nii",
-             "rest_hcp_agg__component_ica_.nii"]
+    opts = {'uncompress': True}
+    files = ["ALL_HC_unthresholded_tmaps.nii.gz",
+             "RSN_HC_unthresholded_tmaps.nii.gz",
+             "rest_hcp_agg__component_ica_.nii.gz"]
 
     labels = [('Basal Ganglia', [21]),
               ('Auditory', [17]),
@@ -873,7 +873,7 @@ def fetch_atlas_allen_2011(data_dir=None, url=None, resume=True, verbose=1):
 
     networks = [[name] * len(idxs) for name, idxs in labels]
 
-    filenames = [(f, url + f, opts) for f in files]
+    filenames = [(os.path.join('allen_rsn_2011', f), url, opts) for f in files]
 
     data_dir = _get_dataset_dir(dataset_name, data_dir=data_dir,
                                 verbose=verbose)


Test patch:
diff --git a/nilearn/datasets/tests/test_atlas.py b/nilearn/datasets/tests/test_atlas.py
--- a/nilearn/datasets/tests/test_atlas.py
+++ b/nilearn/datasets/tests/test_atlas.py
@@ -459,13 +459,14 @@ def test_fetch_atlas_allen_2011():
             "rsn28",
             "comps")
 
-    filenames = ["ALL_HC_unthresholded_tmaps.nii",
-                 "RSN_HC_unthresholded_tmaps.nii",
-                 "rest_hcp_agg__component_ica_.nii"]
+    filenames = ["ALL_HC_unthresholded_tmaps.nii.gz",
+                 "RSN_HC_unthresholded_tmaps.nii.gz",
+                 "rest_hcp_agg__component_ica_.nii.gz"]
 
-    assert_equal(len(tst.mock_url_request.urls), 3)
+    assert_equal(len(tst.mock_url_request.urls), 1)
     for key, fn in zip(keys, filenames):
-        assert_equal(bunch[key], os.path.join(tst.tmpdir, 'allen_rsn_2011', fn))
+        assert_equal(bunch[key], os.path.join(tst.tmpdir, 'allen_rsn_2011',
+                                              'allen_rsn_2011', fn))
 
     assert_not_equal(bunch.description, '')