Repository: sunpy/sunpy

Issue:
Support url-tar method downloads in the JSOCClient
<!--
We know asking good questions takes effort, and we appreciate your time.
Thank you.

Please be aware that everyone has to follow our code of conduct:
https://github.com/sunpy/sunpy/blob/master/CODE_OF_CONDUCT.rst

Also that these comments are hidden when you submit this github issue.

Please have a search on our GitHub repository to see if a similar issue has already been posted.
If a similar issue is closed, have a quick look to see if you are satisfied by the resolution.
If not please go ahead and open an issue!
-->

### Description
<!--
Provide a general description of the feature you would like.
If you prefer, you can also suggest a draft design or API.
This way we have a deeper discussion on the feature.
-->

Currently we hard code downloading files from the JSOC using the `url` method. This in general is the nicest way for our users as they get the number of files they expect to get from the search results returned to them, and they can be opened with no further manual steps.

However, sometimes if you are requesting *a lot* of files, this isn't a good situation for the JSOC who have limited resources to service the download request. We should at the very least enable people to manually select url-tar when they need to. Also we should probably document doing that if you are planning on downloading >N files where N is large.

### Additional context
<!--
Add any other context or screenshots
This part is optional.
Delete this section heading if you do not use it.
-->

This request came from the JSOC team as they have had people using "sunpy" trying to download a sufficient number of files they have had to block their IPs.


Discussion:
I've managed to hack my way around this because by forcing method='url-tar' in sunpy.net.jsoc.JSOCClient.request_data but it required manual downloading via provided url from res.urls.url[0] (where res is my return object from JSOCClient.request_data). I mean, its doable, but i didn't have time to make it nice.

I would like to work on this, but i cant guarantee how much time will i have in the next few weeks but i would like to try implementing this. AFAIK, drms is supporting url-tar method, so we just need to use it nicely.
So here is a patch.file 

```python
diff --git a/sunpy/net/jsoc/jsoc.py b/sunpy/net/jsoc/jsoc.py
index 0768614c8..1bfda0f77 100644
--- a/sunpy/net/jsoc/jsoc.py
+++ b/sunpy/net/jsoc/jsoc.py
@@ -410,7 +410,7 @@ class JSOCClient(BaseClient):
 
         return res
 
-    def request_data(self, jsoc_response, **kwargs):
+    def request_data(self, jsoc_response, tar_only=False, **kwargs):
         """
         Request that JSOC stages the data for download. This method will not
         wait for the request to be staged.
@@ -442,8 +442,10 @@ class JSOCClient(BaseClient):
                 error_message = "Protocols other than fits and as-is are "\
                                 "are not supported."
                 raise TypeError(error_message)
-
-            method = 'url' if protocol == 'fits' else 'url_quick'
+            if tar_only:
+                method = 'url-tar'
+            else:
+                method = 'url' if protocol == 'fits' else 'url_quick'
             r = cd.export(ds, method=method, protocol=protocol)
 
             requests.append(r)
@@ -589,6 +591,8 @@ class JSOCClient(BaseClient):
 
         paths = []
         for request in requests:
+            if request.method == 'url-tar':
+                paths.append(Path(request.tarfile).name)
             for filename in request.data['filename']:
                 # Ensure we don't duplicate the file extension
                 ext = os.path.splitext(filename)[1]
@@ -616,7 +620,7 @@ class JSOCClient(BaseClient):
         urls = []
         for request in requests:
             if request.status == 0:
-                if request.protocol == 'as-is':
+                if request.protocol == 'as-is' or request.method == 'url-tar':
                     urls.extend(list(request.urls.url))
                 else:
                     for index, data in request.data.iterrows():
```

I have minimal working example compatible with jsoc without making to much of a mess. It works if you do something like

```python
imports..
client = jsoc.JSOCClient()
resjsoc = client.search(some_query)
res = client.request_data(resjsoc, tar_only=True)
In [15]: res.data                                                                                                                                                                                                                             
Out[15]: 
                                                record                                           filename
0     hmi.sharp_cea_720s[114][2010.08.03_05:48:00_TAI]  hmi.sharp_cea_720s.114.20100803_054800_TAI.con...
1     hmi.sharp_cea_720s[114][2010.08.03_05:48:00_TAI]  hmi.sharp_cea_720s.114.20100803_054800_TAI.Bp....
2     hmi.sharp_cea_720s[114][2010.08.03_05:48:00_TAI]  hmi.sharp_cea_720s.114.20100803_054800_TAI.Bt....
3     hmi.sharp_cea_720s[114][2010.08.03_05:48:00_TAI]  hmi.sharp_cea_720s.114.20100803_054800_TAI.Br....
4     hmi.sharp_cea_720s[114][2010.08.03_06:00:00_TAI]  hmi.sharp_cea_720s.114.20100803_060000_TAI.con...
...                                                ...                                                ...
4991  hmi.sharp_cea_720s[114][2010.08.13_15:36:00_TAI]  hmi.sharp_cea_720s.114.20100813_153600_TAI.Br....
4992  hmi.sharp_cea_720s[114][2010.08.13_15:48:00_TAI]  hmi.sharp_cea_720s.114.20100813_154800_TAI.con...
4993  hmi.sharp_cea_720s[114][2010.08.13_15:48:00_TAI]  hmi.sharp_cea_720s.114.20100813_154800_TAI.Bp....
4994  hmi.sharp_cea_720s[114][2010.08.13_15:48:00_TAI]  hmi.sharp_cea_720s.114.20100813_154800_TAI.Bt....
4995  hmi.sharp_cea_720s[114][2010.08.13_15:48:00_TAI]  hmi.sharp_cea_720s.114.20100813_154800_TAI.Br....

[4996 rows x 2 columns]


In [22]: res.urls.url                                                                                                                                                                                                                         
Out[22]: 
0    http://jsoc.stanford.edu/SUM66/D1313302623/S00...
Name: url, dtype: object

In [23]: down_stuff = client.get_request(res)                                                                                                                                                                                                 
1 URLs found for download. Full request totalling 1732MB
Files Downloaded: 100%|██████████████████████████████████████████████████████████████| 1/1 [00:00<00:00,  2.89file/s]

```

So i just want to confirm if you guys are ok with this approach. If ok, i will create PR and try to write some test for it. If not, please tell me so i can change how this is implemented.


Resolving patch:
diff --git a/sunpy/net/jsoc/jsoc.py b/sunpy/net/jsoc/jsoc.py
--- a/sunpy/net/jsoc/jsoc.py
+++ b/sunpy/net/jsoc/jsoc.py
@@ -410,7 +410,7 @@ def search_metadata(self, *query, **kwargs):
 
         return res
 
-    def request_data(self, jsoc_response, **kwargs):
+    def request_data(self, jsoc_response, method='url', **kwargs):
         """
         Request that JSOC stages the data for download. This method will not
         wait for the request to be staged.
@@ -420,6 +420,13 @@ def request_data(self, jsoc_response, **kwargs):
         jsoc_response : `~sunpy.net.jsoc.jsoc.JSOCResponse` object
             The results of a query
 
+        method : {'url', 'url-tar', 'url-quick'}
+            Method for requesting JSOC data, can be 'url-tar', 'url' (the default) and 'url-quick'
+            If 'url-tar' it will request JSOC to provide single .tar file which contains all data
+            If 'url' it will request JSOC to provide all data as separate .fits files
+            If 'url-quick' (only with protocol 'as-is') provide all data as separate files,
+            but only if data is online.
+
         Returns
         -------
         requests : `~drms.ExportRequest` object or
@@ -432,18 +439,25 @@ def request_data(self, jsoc_response, **kwargs):
 
         requests = []
         self.query_args = jsoc_response.query_args
+        supported_protocols = {'fits', 'as-is'}
+        supported_methods = {'url-tar', 'url', 'url-quick'}
         for block in jsoc_response.query_args:
 
             ds = self._make_recordset(**block)
             cd = drms.Client(email=block.get('notify', ''))
             protocol = block.get('protocol', 'fits')
 
-            if protocol != 'fits' and protocol != 'as-is':
-                error_message = "Protocols other than fits and as-is are "\
+            if protocol not in supported_protocols:
+                error_message = f"Protocols other than {','.join(supported_protocols)} "\
+                                "are not supported."
+                raise TypeError(error_message)
+            if method not in supported_methods:
+                error_message = f"Methods other than {','.join(supported_methods)} "\
                                 "are not supported."
                 raise TypeError(error_message)
 
-            method = 'url' if protocol == 'fits' else 'url_quick'
+            if method != 'url-tar':
+                method = 'url' if protocol == 'fits' else 'url_quick'
             r = cd.export(ds, method=method, protocol=protocol)
 
             requests.append(r)
@@ -589,16 +603,20 @@ def get_request(self, requests, path=None, overwrite=False, progress=True,
 
         paths = []
         for request in requests:
-            for filename in request.data['filename']:
-                # Ensure we don't duplicate the file extension
-                ext = os.path.splitext(filename)[1]
-                if path.endswith(ext):
-                    fname = path.strip(ext)
-                else:
-                    fname = path
-                fname = fname.format(file=filename)
-                fname = os.path.expanduser(fname)
-                paths.append(fname)
+            if request.method == 'url-tar':
+                fname = path.format(file=Path(request.tarfile).name)
+                paths.append(os.path.expanduser(fname))
+            else:
+                for filename in request.data['filename']:
+                    # Ensure we don't duplicate the file extension
+                    ext = os.path.splitext(filename)[1]
+                    if path.endswith(ext):
+                        fname = path.strip(ext)
+                    else:
+                        fname = path
+                    fname = fname.format(file=filename)
+                    fname = os.path.expanduser(fname)
+                    paths.append(fname)
 
         if max_conn * kwargs['max_splits'] > 10:
             warnings.warn(("JSOC does not support more than 10 parallel connections. " +
@@ -616,7 +634,7 @@ def get_request(self, requests, path=None, overwrite=False, progress=True,
         urls = []
         for request in requests:
             if request.status == 0:
-                if request.protocol == 'as-is':
+                if request.protocol == 'as-is' or request.method == 'url-tar':
                     urls.extend(list(request.urls.url))
                 else:
                     for index, data in request.data.iterrows():


Test patch:
diff --git a/sunpy/net/jsoc/tests/test_jsoc.py b/sunpy/net/jsoc/tests/test_jsoc.py
--- a/sunpy/net/jsoc/tests/test_jsoc.py
+++ b/sunpy/net/jsoc/tests/test_jsoc.py
@@ -146,13 +146,34 @@ def test_get_request(client):
     responses = client.search(
         a.Time('2012/1/1T1:00:36', '2012/1/1T01:00:38'),
         a.jsoc.Series('hmi.M_45s'), a.jsoc.Notify('jsoc@cadair.com'))
-
     bb = client.request_data(responses)
     path = tempfile.mkdtemp()
     aa = client.get_request(bb, path=path)
     assert isinstance(aa, Results)
 
 
+@pytest.mark.remote_data
+def test_get_request_tar(client):
+    responses = client.search(
+        a.Time('2012/1/1T1:00:36', '2012/1/1T01:00:38'),
+        a.jsoc.Series('hmi.M_45s'), a.jsoc.Notify('jsoc@cadair.com'))
+    bb = client.request_data(responses, method='url-tar')
+    bb.wait()
+    path = tempfile.mkdtemp()
+    aa = client.get_request(bb, path=path)
+    assert isinstance(aa, Results)
+
+    responses = client.search(
+        a.Time('2012/1/1T1:00:36', '2012/1/1T01:00:38'),
+        a.jsoc.Series('hmi.M_45s'), a.jsoc.Notify('jsoc@cadair.com'),
+        a.jsoc.Protocol('as-is'))
+    bb = client.request_data(responses, method='url-tar')
+    bb.wait()
+    path = tempfile.mkdtemp()
+    aa = client.get_request(bb, path=path)
+    assert isinstance(aa, Results)
+
+
 @pytest.mark.remote_data
 def test_invalid_query(client):
     with pytest.raises(ValueError):
@@ -319,6 +340,26 @@ def test_request_data_protocol(client):
     assert req._d['protocol'] == 'as-is'
 
 
+@pytest.mark.remote_data
+def test_request_data_method(client):
+    responses = client.search(
+        a.Time('2012/1/1T1:00:36', '2012/1/1T01:00:38'),
+        a.jsoc.Series('hmi.M_45s'), a.jsoc.Notify('jsoc@cadair.com'))
+    req = client.request_data(responses, method='url-tar')
+    req.wait()
+    assert req._d['method'] == 'url-tar'
+    assert req._d['protocol'] == 'fits'
+
+    responses = client.search(
+        a.Time('2012/1/1T1:00:36', '2012/1/1T01:00:38'),
+        a.jsoc.Series('hmi.M_45s'), a.jsoc.Notify('jsoc@cadair.com'),
+        a.jsoc.Protocol('as-is'))
+    req = client.request_data(responses, method='url-tar')
+    req.wait()
+    assert req._d['method'] == 'url-tar'
+    assert req._d['protocol'] == 'as-is'
+
+
 @pytest.mark.remote_data
 def test_check_request(client):
     responses = client.search(