Repository: sunpy/sunpy

Issue:
Mangled filename when downloading MDI data from VSO using Fido
<!-- This comments are hidden when you submit the issue so you do not need to remove them!
Please be sure to check out our contributing guidelines: https://github.com/sunpy/sunpy/blob/master/CONTRIBUTING.rst
Please be sure to check out our code of conduct:
https://github.com/sunpy/sunpy/blob/master/CODE_OF_CONDUCT.rst -->

<!-- Please have a search on our GitHub repository to see if a similar issue has already been posted.
If a similar issue is closed, have a quick look to see if you are satisfied by the resolution.
If not please go ahead and open an issue! -->

### Description
<!-- Provide a general description of the bug. -->
When downloading MDI data from VSO the files are saved with incorrect filenames, parts of the content-encoding header are being included in the file name. It looks like the headers are not being set correctly on the response sent from VSO but this may not be easily fixed so a work around may be required.

### Expected behavior
<!-- What did you expect to happen. -->
Files to be saved with the correct filename

### Actual behavior
<!-- What actually happened. -->
<!-- Was the output confusing or poorly described? -->
Files are not save with the correct filename


### Steps to Reproduce
<!-- Ideally a code example could be provided so we can run it ourselves. -->
<!-- If you are pasting code, use tripe backticks (```) around your code snippet. -->

```python
results = Fido.search(a.Time('2011/1/1 01:00', '2011/1/1 01:02'), a.Instrument(hmii') | a.Instrument('mdi'))
files = Fido.fetch(results[:,0])
print(files)
['/Users/shane/sunpy/data/hmi_ic_45s_2011_01_01_01_01_30_tai_continuum.fits', '/Users/shane/sunpy/data/mdi_vw_v_9466622_9466622.tar"\\nContent-transfer-encoding: binary']
```
### System Details
<!-- We at least need to know the SunPy version you are using. -->
<!-- We provide a short function in SunPy that will provide some of the below information. -->
<!-- It is sunpy.util.system_info(), this is optional but strongly recommended. -->

  - SunPy Version: 1.0.2
  - Astropy Version: 3.2.1 
  - Python Version: 3.7.0
  - OS information: macOS 10.14.6
Mangled filename when downloading MDI data from VSO using Fido
<!-- This comments are hidden when you submit the issue so you do not need to remove them!
Please be sure to check out our contributing guidelines: https://github.com/sunpy/sunpy/blob/master/CONTRIBUTING.rst
Please be sure to check out our code of conduct:
https://github.com/sunpy/sunpy/blob/master/CODE_OF_CONDUCT.rst -->

<!-- Please have a search on our GitHub repository to see if a similar issue has already been posted.
If a similar issue is closed, have a quick look to see if you are satisfied by the resolution.
If not please go ahead and open an issue! -->

### Description
<!-- Provide a general description of the bug. -->
When downloading MDI data from VSO the files are saved with incorrect filenames, parts of the content-encoding header are being included in the file name. It looks like the headers are not being set correctly on the response sent from VSO but this may not be easily fixed so a work around may be required.

### Expected behavior
<!-- What did you expect to happen. -->
Files to be saved with the correct filename

### Actual behavior
<!-- What actually happened. -->
<!-- Was the output confusing or poorly described? -->
Files are not save with the correct filename


### Steps to Reproduce
<!-- Ideally a code example could be provided so we can run it ourselves. -->
<!-- If you are pasting code, use tripe backticks (```) around your code snippet. -->

```python
results = Fido.search(a.Time('2011/1/1 01:00', '2011/1/1 01:02'), a.Instrument(hmii') | a.Instrument('mdi'))
files = Fido.fetch(results[:,0])
print(files)
['/Users/shane/sunpy/data/hmi_ic_45s_2011_01_01_01_01_30_tai_continuum.fits', '/Users/shane/sunpy/data/mdi_vw_v_9466622_9466622.tar"\\nContent-transfer-encoding: binary']
```
### System Details
<!-- We at least need to know the SunPy version you are using. -->
<!-- We provide a short function in SunPy that will provide some of the below information. -->
<!-- It is sunpy.util.system_info(), this is optional but strongly recommended. -->

  - SunPy Version: 1.0.2
  - Astropy Version: 3.2.1 
  - Python Version: 3.7.0
  - OS information: macOS 10.14.6


Discussion:
This is really a bug with the VSO, when doing the http request to download the MDI file you get the following response:

```
$ http -v GET "http://vso3.stanford.edu/cgi-bin/drms-export-sha.cgi?series=mdi__fd_M_96m_lev182;compress=rice;record=98611-98611"                                                                                                                                                          
GET /cgi-bin/drms-export-sha.cgi?series=mdi__fd_M_96m_lev182;compress=rice;record=98611-98611 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: vso3.stanford.edu
User-Agent: HTTPie/1.0.3



HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Disposition: attachment; filename="mdi_fd_M_96m_lev182_98611_98611.tar"\nContent-transfer-encoding: binary
Content-Type: application/octet-stream
Date: Thu, 26 Sep 2019 10:15:39 GMT
Keep-Alive: timeout=5, max=100
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 mod_nss/1.0.14 NSS/3.28.4 mod_wsgi/3.4 Python/2.7.5
Transfer-Encoding: chunked
```

The issue is with the content disposition header:

```
Content-Disposition: attachment; filename="mdi_fd_M_96m_lev182_98611_98611.tar"\nContent-transfer-encoding: binary
```

This is an invalid content disposition header (as far as I can tell by reading [RFC 6266](https://tools.ietf.org/html/rfc6266). [parfive](https://github.com/Cadair/parfive/blob/master/parfive/utils.py#L31) uses the stdlib [`cgi.parse_header`](https://docs.python.org/3.7/library/cgi.html#cgi.parse_header) function to extract the filename from the header. I am assuming that that (very old) bit of the stdlib is pretty robust and correct.

I have reported this to the VSO folk, who are looking into fixing it.
Another option might be to handle this in the vso client code specifically in`mk_filename` function in this area https://github.com/sunpy/sunpy/blob/558707fd840c47ce9f6599288b37d1aa2ed54c60/sunpy/net/vso/vso.py#L412-L419

which seems to be passed to parfive here

https://github.com/sunpy/sunpy/blob/558707fd840c47ce9f6599288b37d1aa2ed54c60/sunpy/net/vso/vso.py#L840-L843
huh, yeah that's where it would need fixing, how come we didn't switch that to parfive I wonder. I am certainly happier about fixing it specifically in the VSO code.
We don't have to use VSO to fetch MDI data; we can use JSOC (in the same way we use it to fetch HMI and AIA data). The path in the error message that includes `series=mdi__fd_M_96m_lev182` is referring to the drms series `mdi.fd_M_96m_lev182`.

Edit: Also @samaloney you can use the `drms` package to download MDI data from JSOC.
Would be interesting to double check this issue isn't present on JSOC as well, according to the VSO team the issue is in the DRMS C code.
I don't think JSOC has any issues at the moment. I tried using the web interface as well as the `drms` package and both worked fine for me. @samaloney I am not exactly sure what time you are querying so I picked a random time in the code block below, which works for me (with the correct e-mail address): 

```
import drms
c = drms.Client(email='person@email.address', verbose=True)
keys, segments = c.query('mdi.fd_M_96m_lev182[2010.04.17/24h]', key='T_REC', seg='data')
segments.data[0]
mdidata = c.export('mdi.fd_M_96m_lev182[2010.04.17/24h]{data}')
mdidata.urls.url[0]
```
You can also do this (arguably :wink:) with a nicer interface with Fido:
```python
Fido.search(a.Time("2010/04/17 00:00", "2010/04/18 00:00"), a.Sample(24*u.hour), a.jsoc.Series("mdi.fd_M_96m_lev182"), a.jsoc.Notify("person@email.address"))
```

@mbobra your example doesn't result in a URL which specifies a filename at all, so definitely isn't subject to the bug :rofl: 
@Cadair Yes for sure, not trying to bash Fido 😃Glad the example checks out.
I know, both do the job, just differently lol.
@mbobra thanks I could have gotten the data if I really wanted to but I was just trying to get some MDI and HMI files for a quick test and the above _should_ have worked which is the issue.

@Cadair actually why doesn't Fido query JSOC as well as VSO for this as both should be able to handle this query?

A bigger question would be if it did, which would take precedence or how would this be handled.

I'm back in the office this week so should have some time to look at this again.
The problem still persists in VSO and latest Sunpy.

Got this kind of filename: ```'mdi_fd_m_96m_lev182_39633_39633.tar"\nContent-transfer-encoding: binary'```
A generic algorithm for fixing this is [here](https://github.com/Cadair/parfive/pull/28/files#diff-cf9ef7a01d30e4debe27a090444b5368L29-L36). However, the way we are currently determining the filename for the VSO download is unclear (in `mk_filename` in vso.py) so I have asked the VSO team for some advice on what the best algorithm is.
This is really a bug with the VSO, when doing the http request to download the MDI file you get the following response:

```
$ http -v GET "http://vso3.stanford.edu/cgi-bin/drms-export-sha.cgi?series=mdi__fd_M_96m_lev182;compress=rice;record=98611-98611"                                                                                                                                                          
GET /cgi-bin/drms-export-sha.cgi?series=mdi__fd_M_96m_lev182;compress=rice;record=98611-98611 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate
Connection: keep-alive
Host: vso3.stanford.edu
User-Agent: HTTPie/1.0.3



HTTP/1.1 200 OK
Connection: Keep-Alive
Content-Disposition: attachment; filename="mdi_fd_M_96m_lev182_98611_98611.tar"\nContent-transfer-encoding: binary
Content-Type: application/octet-stream
Date: Thu, 26 Sep 2019 10:15:39 GMT
Keep-Alive: timeout=5, max=100
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_fcgid/2.3.9 mod_nss/1.0.14 NSS/3.28.4 mod_wsgi/3.4 Python/2.7.5
Transfer-Encoding: chunked
```

The issue is with the content disposition header:

```
Content-Disposition: attachment; filename="mdi_fd_M_96m_lev182_98611_98611.tar"\nContent-transfer-encoding: binary
```

This is an invalid content disposition header (as far as I can tell by reading [RFC 6266](https://tools.ietf.org/html/rfc6266). [parfive](https://github.com/Cadair/parfive/blob/master/parfive/utils.py#L31) uses the stdlib [`cgi.parse_header`](https://docs.python.org/3.7/library/cgi.html#cgi.parse_header) function to extract the filename from the header. I am assuming that that (very old) bit of the stdlib is pretty robust and correct.

I have reported this to the VSO folk, who are looking into fixing it.
Another option might be to handle this in the vso client code specifically in`mk_filename` function in this area https://github.com/sunpy/sunpy/blob/558707fd840c47ce9f6599288b37d1aa2ed54c60/sunpy/net/vso/vso.py#L412-L419

which seems to be passed to parfive here

https://github.com/sunpy/sunpy/blob/558707fd840c47ce9f6599288b37d1aa2ed54c60/sunpy/net/vso/vso.py#L840-L843
huh, yeah that's where it would need fixing, how come we didn't switch that to parfive I wonder. I am certainly happier about fixing it specifically in the VSO code.
We don't have to use VSO to fetch MDI data; we can use JSOC (in the same way we use it to fetch HMI and AIA data). The path in the error message that includes `series=mdi__fd_M_96m_lev182` is referring to the drms series `mdi.fd_M_96m_lev182`.

Edit: Also @samaloney you can use the `drms` package to download MDI data from JSOC.
Would be interesting to double check this issue isn't present on JSOC as well, according to the VSO team the issue is in the DRMS C code.
I don't think JSOC has any issues at the moment. I tried using the web interface as well as the `drms` package and both worked fine for me. @samaloney I am not exactly sure what time you are querying so I picked a random time in the code block below, which works for me (with the correct e-mail address): 

```
import drms
c = drms.Client(email='person@email.address', verbose=True)
keys, segments = c.query('mdi.fd_M_96m_lev182[2010.04.17/24h]', key='T_REC', seg='data')
segments.data[0]
mdidata = c.export('mdi.fd_M_96m_lev182[2010.04.17/24h]{data}')
mdidata.urls.url[0]
```
You can also do this (arguably :wink:) with a nicer interface with Fido:
```python
Fido.search(a.Time("2010/04/17 00:00", "2010/04/18 00:00"), a.Sample(24*u.hour), a.jsoc.Series("mdi.fd_M_96m_lev182"), a.jsoc.Notify("person@email.address"))
```

@mbobra your example doesn't result in a URL which specifies a filename at all, so definitely isn't subject to the bug :rofl: 
@Cadair Yes for sure, not trying to bash Fido 😃Glad the example checks out.
I know, both do the job, just differently lol.
@mbobra thanks I could have gotten the data if I really wanted to but I was just trying to get some MDI and HMI files for a quick test and the above _should_ have worked which is the issue.

@Cadair actually why doesn't Fido query JSOC as well as VSO for this as both should be able to handle this query?

A bigger question would be if it did, which would take precedence or how would this be handled.

I'm back in the office this week so should have some time to look at this again.
The problem still persists in VSO and latest Sunpy.

Got this kind of filename: ```'mdi_fd_m_96m_lev182_39633_39633.tar"\nContent-transfer-encoding: binary'```
A generic algorithm for fixing this is [here](https://github.com/Cadair/parfive/pull/28/files#diff-cf9ef7a01d30e4debe27a090444b5368L29-L36). However, the way we are currently determining the filename for the VSO download is unclear (in `mk_filename` in vso.py) so I have asked the VSO team for some advice on what the best algorithm is.

Resolving patch:
diff --git a/sunpy/net/vso/vso.py b/sunpy/net/vso/vso.py
--- a/sunpy/net/vso/vso.py
+++ b/sunpy/net/vso/vso.py
@@ -5,8 +5,10 @@
 
 import os
 import re
+import cgi
 import sys
 import socket
+import datetime
 import warnings
 import itertools
 from functools import partial
@@ -15,21 +17,21 @@
 from urllib.request import urlopen
 
 import zeep
+from parfive import Downloader, Results
 from zeep.helpers import serialize_object
 
 import astropy.units as u
 from astropy.table import QTable as Table
-from parfive import Downloader, Results
 
 from sunpy import config
-from sunpy.time import TimeRange, parse_time
-from sunpy.net.vso import attrs
 from sunpy.net.attr import and_
-from sunpy.util.net import slugify, get_content_disposition
-from sunpy.net.vso.attrs import TIMEFORMAT, walker
 from sunpy.net.base_client import BaseClient
+from sunpy.net.vso import attrs
+from sunpy.net.vso.attrs import TIMEFORMAT, walker
+from sunpy.time import TimeRange, parse_time
 from sunpy.util.decorators import deprecated
 from sunpy.util.exceptions import SunpyUserWarning
+from sunpy.util.net import get_content_disposition, slugify
 
 from .zeep_plugins import SunPyLoggingZeepPlugin
 
@@ -414,26 +416,51 @@ def merge(self, queryresponses):
 
     @staticmethod
     def mk_filename(pattern, queryresponse, resp, url):
+        """
+        Generate the best possible (or least-worse) filename for a VSO download.
+
+        * Use the ``content-disposition`` header.
+        * Use `fileid` to generate a file name if content-disposition fails
+        * If everything else fails use the last segment of the URL and hope.
+        """
         name = None
-        url_filename = url.split('/')[-1]
         if resp:
-            name = resp.headers.get("Content-Disposition", url_filename)
-            if name:
-                name = get_content_disposition(name)
-        if not name:
+            cdheader = resp.headers.get("Content-Disposition", None)
+            if cdheader:
+                value, params = cgi.parse_header(cdheader)
+                name = params.get('filename', "")
+                # Work around https://github.com/sunpy/sunpy/issues/3372
+                if name.count('"') >= 2:
+                    name = name.split('"')[1]
+
+        if name is None:
+            # Advice from the VSO is to fallback to providerid + fileid
+            # As it's possible multiple providers give the same fileid.
+            # However, I haven't implemented this yet as it would be a breaking
+            # change to the filenames we expect.
+
+            # I don't know if we still need this bytes check in Python 3 only
+            # land, but I don't dare remove it.
             if isinstance(queryresponse.fileid, bytes):
-                name = queryresponse.fileid.decode("ascii", "ignore")
+                fileid = queryresponse.fileid.decode("ascii", "ignore")
             else:
-                name = queryresponse.fileid
+                fileid = queryresponse.fileid
+
+            # Some providers make fileid a path
+            # Some also don't specify a file extension, but not a lot we can do
+            # about that.
+            name = fileid.split("/")[-1]
 
-        fs_encoding = sys.getfilesystemencoding()
-        if fs_encoding is None:
-            fs_encoding = "ascii"
+        # If somehow we have got this far with an empty string, fallback to url segment
+        if not name:
+            name = url.split('/')[-1]
 
+        # Remove any not-filename appropriate characters
         name = slugify(name)
 
+        # If absolutely everything else fails make a filename based on download time
         if not name:
-            name = "file"
+            name = f"vso_file_{datetime.datetime.now().strftime('%Y%m%d%H%M%S%f')}"
 
         fname = pattern.format(file=name, **serialize_object(queryresponse))
 


Test patch:
diff --git a/sunpy/net/tests/test_vso.py b/sunpy/net/tests/test_vso.py
--- a/sunpy/net/tests/test_vso.py
+++ b/sunpy/net/tests/test_vso.py
@@ -475,3 +475,15 @@ def test_vso_error(client):
         client.search(
             va.Time('2019/12/30', '2019/12/31'),
             va.Instrument('ovsa'))
+
+
+@pytest.mark.remote_data
+def test_incorrect_content_disposition(client):
+    results = client.search(
+        va.Time('2011/1/1 01:00', '2011/1/1 01:02'),
+        va.Instrument('mdi'))
+    files = client.fetch(results[0:1])
+
+    assert len(files) == 1
+    assert  files[0].endswith("mdi_vw_v_9466622_9466622.tar")
+    assert "Content" not in files[0]