Repository: sunpy/sunpy

Issue:
Scraper errors when directory in pattern doesn't exist
i.e. when you have a pattern like `http://mywebsite/%Y/%Y%m%d` if you ask for a year that doesn't exist it will fail because it will try and get the directory.
Transient test failure in scraper code
This seems to happen in Travis builds from time to time.

```python
=================================== FAILURES ===================================
_________________________ testURL_patternMilliseconds __________________________
    def testURL_patternMilliseconds():
        s = Scraper('fd_%Y%m%d_%H%M%S_%e.fts')
>       assert s._URL_followsPattern('fd_20130410_231211_119.fts')
E       assert False
E        +  where False = <bound method Scraper._URL_followsPattern of <sunpy.util.scraper.Scraper object at 0x7fdfda7c0400>>('fd_20130410_231211_119.fts')
E        +    where <bound method Scraper._URL_followsPattern of <sunpy.util.scraper.Scraper object at 0x7fdfda7c0400>> = <sunpy.util.scraper.Scraper object at 0x7fdfda7c0400>._URL_followsPattern
sunpy/util/tests/test_scraper.py:110: AssertionError
============================= 121 tests deselected =============================
============ 1 failed, 935 passed, 121 deselected in 62.03 seconds =============
```


Discussion:
```
Traceback (most recent call last):
  File "/Users/brst3037/opt/anaconda36/envs/sunpy/lib/python3.6/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<input>", line 1, in <module>
  File "/Users/brst3037/code/sunpy/sunpy/net/fido_factory.py", line 326, in search
    return UnifiedResponse(query_walker.create(query, self))
  File "/Users/brst3037/code/sunpy/sunpy/net/attr.py", line 240, in create
    return self.createmm(self, *args, **kwargs)
  File "/Users/brst3037/code/sunpy/sunpy/util/multimethod.py", line 120, in __call__
    return cached(*args, **kwargs)
  File "/Users/brst3037/code/sunpy/sunpy/net/fido_factory.py", line 260, in _create_and
    return [factory._make_query_to_client(*query.attrs)]
  File "/Users/brst3037/code/sunpy/sunpy/net/fido_factory.py", line 418, in _make_query_to_client
    return tmpclient.search(*query), tmpclient
  File "/Users/brst3037/code/sunpy/sunpy/net/dataretriever/client.py", line 324, in search
    self.map_.get('TimeRange'), **kwergs)
  File "/Users/brst3037/code/sunpy/sunpy/net/dataretriever/sources/eve.py", line 170, in _get_url_for_timerange
    return eve.filelist(timerange)
  File "/Users/brst3037/code/sunpy/sunpy/util/scraper.py", line 226, in filelist
    opn = urlopen(directory)
  File "/Users/brst3037/opt/anaconda36/envs/sunpy/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/Users/brst3037/opt/anaconda36/envs/sunpy/lib/python3.6/urllib/request.py", line 532, in open
    response = meth(req, response)
  File "/Users/brst3037/opt/anaconda36/envs/sunpy/lib/python3.6/urllib/request.py", line 642, in http_response
    'http', request, response, code, msg, hdrs)
  File "/Users/brst3037/opt/anaconda36/envs/sunpy/lib/python3.6/urllib/request.py", line 570, in error
    return self._call_chain(*args)
  File "/Users/brst3037/opt/anaconda36/envs/sunpy/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/Users/brst3037/opt/anaconda36/envs/sunpy/lib/python3.6/urllib/request.py", line 650, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found
```
What's the expected fix, return a second list of failed files or include the failed ones in the currently returned list? 
@Cadair your thoughts? 
I would need to double check how the scraper handles it, but I think the answer is just to return an empty list because it dosen't find any files.
looks like @Brhstone has a patch here: https://github.com/Brhstone/sunpy/commit/a62eec76d527a91101dc35a01284148122a47aec#diff-587d81658a6b5d15d91bb0e994a9f63b
How would you then know what data wasn't found, I was going to suggest just return the http status error code instead of the url?
The scraper only returns results for files it can find on the remote host based on a URL pattern. If you want to know all possible urls based on the pattern you will probably have to generate them yourself.
Ok. That codes not in a pull request so what the procedure, clone that repo and make  PR for that one change?
Ask @Brhstone nicely to PR the fix separately :laughing: 

Failing that I think you will have to just remake the patch because there isn't really a commit you can cherry-pick
The way the scraper works I would expect to only get a list with the available files. So, if my query ranges 3 days, but there's no data in the middle one, the list of files should only show the ones on the first and last day. Non-error should be shown as the scraper doesn't have a concept of what's the expected cadence (though it could be added and then show some stats or warnings). The only errors I expect is when the user doesn't have a working connection, the server is not available or when you don't have permissions to access to these directories (as it could be an ftp directory that needs the appropriate credentials).

Since we don't know whether the server is supposed to have that data or not... some instruments produce data constantly from space, but ground-based telescopes may have not have observations because it was raining... Or they are not updating the data in the server every day. So, showing the 404 error message is not more helpful to the user than just giving a blank list. It's up to the user to check again later or investigate why there's no data that day.
@Cadair, thanks for the reminder. ;)

I should be able to make that PR tomorrow.
Also these two online tests suddenly broke:
```python
=================================== FAILURES ===================================
_____________________ testFilesRange_sameDirectory_remote ______________________
    @pytest.mark.online
    def testFilesRange_sameDirectory_remote():
        pattern = ('http://solarmonitor.org/data/%Y/%m/%d/'
                   'fits/{instrument}/'
                   '{instrument}_00174_fd_%Y%m%d_%H%M%S.fts.gz')
        s = Scraper(pattern, instrument = 'swap')
        startdate = datetime.datetime(2014, 5, 14, 0, 0)
        enddate = datetime.datetime(2014, 5, 14, 6, 30)
        timerange = TimeRange(startdate, enddate)
>       assert len(s.filelist(timerange)) == 2
E       assert 0 == 2
E        +  where 0 = len([])
E        +    where [] = <bound method Scraper.filelist of <sunpy.util.scraper.Scraper object at 0x7f441e7001d0>>(    Start: 2014-05-14 00:00:00\n    End:   2014-05-14 06:30:00\n    Center:2014-05-14 03:15:00\n    Duration:0.2708333333333333 days or\n           6.5 hours or\n           390.0 minutes or\n           23400.0 seconds\n)
E        +      where <bound method Scraper.filelist of <sunpy.util.scraper.Scraper object at 0x7f441e7001d0>> = <sunpy.util.scraper.Scraper object at 0x7f441e7001d0>.filelist
sunpy/util/tests/test_scraper.py:136: AssertionError

__________________ testFilesRange_sameDirectory_months_remote __________________
    @pytest.mark.online
    def testFilesRange_sameDirectory_months_remote():
        pattern = ('http://www.srl.caltech.edu/{spacecraft}/DATA/{instrument}/'
                   'Ahead/1minute/AeH%y%b.1m')
        s = Scraper(pattern, spacecraft='STEREO', instrument = 'HET')
        startdate = datetime.datetime(2007, 8, 1)
        enddate = datetime.datetime(2007, 9, 10)
        timerange = TimeRange(startdate, enddate)
>       assert len(s.filelist(timerange)) == 2
E       assert 0 == 2
E        +  where 0 = len([])
E        +    where [] = <bound method Scraper.filelist of <sunpy.util.scraper.Scraper object at 0x7f441e718e80>>(    Start: 2007-08-01 00:00:00\n    End:   2007-09-10 00:00:00\n    Center:2007-08-21 00:00:00\n    Duration:40.0 days or\n           960.0 hours or\n           57600.0 minutes or\n           3456000.0 seconds\n)
E        +      where <bound method Scraper.filelist of <sunpy.util.scraper.Scraper object at 0x7f441e718e80>> = <sunpy.util.scraper.Scraper object at 0x7f441e718e80>.filelist
sunpy/util/tests/test_scraper.py:150: AssertionError
```
I have xfailed all of these at the moment to get the CI back. (#1953)
I'll try to look into it soon...

Resolving patch:
diff --git a/sunpy/util/scraper.py b/sunpy/util/scraper.py
--- a/sunpy/util/scraper.py
+++ b/sunpy/util/scraper.py
@@ -5,6 +5,7 @@
 import re
 import datetime
 from ftplib import FTP
+from urllib.error import HTTPError
 from urllib.request import urlopen
 
 from bs4 import BeautifulSoup
@@ -67,9 +68,11 @@ def __init__(self, pattern, **kwargs):
         else:
             now = datetime.datetime.now()
             milliseconds_ = int(now.microsecond / 1000.)
-            self.now = now.strftime(self.pattern[0:milliseconds.start()] +
-                                    str(milliseconds_) +
-                                    self.pattern[milliseconds.end():])
+            self.now = now.strftime('{start}{milli:03d}{end}'.format(
+                start=self.pattern[0:milliseconds.start()],
+                milli=milliseconds_,
+                end=self.pattern[milliseconds.end():]
+            ))
 
     def matches(self, filepath, date):
         return date.strftime(self.pattern) == filepath
@@ -234,6 +237,11 @@ def filelist(self, timerange):
                                     filesurls.append(fullpath)
                 finally:
                     opn.close()
+            except HTTPError as http_err:
+                # Ignore missing directories (issue #2684).
+                if http_err.code == 404:
+                    continue
+                raise
             except Exception:
                 raise
         return filesurls


Test patch:
diff --git a/sunpy/util/tests/test_scraper.py b/sunpy/util/tests/test_scraper.py
--- a/sunpy/util/tests/test_scraper.py
+++ b/sunpy/util/tests/test_scraper.py
@@ -1,3 +1,6 @@
+import datetime
+from unittest.mock import patch, Mock
+
 import pytest
 
 import astropy.units as u
@@ -140,22 +143,29 @@ def testURL_pattern():
     assert not s._URL_followsPattern('fd_20130410_ar_231211.fts.gz')
 
 
-@pytest.mark.xfail
-def testURL_patternMilliseconds():
+def testURL_patternMillisecondsGeneric():
     s = Scraper('fd_%Y%m%d_%H%M%S_%e.fts')
-    # NOTE: Seems that if below fails randomly - not understood why
-    #       with `== True` fails a bit less...
     assert s._URL_followsPattern('fd_20130410_231211_119.fts')
     assert not s._URL_followsPattern('fd_20130410_231211.fts.gz')
     assert not s._URL_followsPattern('fd_20130410_ar_231211.fts.gz')
 
 
+def testURL_patternMillisecondsZeroPadded():
+    # Asserts solution to ticket #1954.
+    # Milliseconds must be zero-padded in order to match URL lengths.
+    now_mock = Mock(return_value=datetime.datetime(2019, 4, 19, 0, 0, 0, 4009))
+    with patch('datetime.datetime', now=now_mock):
+        s = Scraper('fd_%Y%m%d_%H%M%S_%e.fts')
+    now_mock.assert_called_once()
+    assert s.now == 'fd_20190419_000000_004.fts'
+
+
 @pytest.mark.xfail
 def testFilesRange_sameDirectory_local():
+    # Fails due to an IsADirectoryError, wrapped in a URLError, after `requests`
+    # tries to open a directory as a binary file.
     s = Scraper('/'.join(['file:/', rootdir,
                           'EIT', 'efz%Y%m%d.%H%M%S_s.fits']))
-    print(s.pattern)
-    print(s.now)
     startdate = parse_time((2004, 3, 1, 4, 0))
     enddate = parse_time((2004, 3, 1, 6, 30))
     assert len(s.filelist(TimeRange(startdate, enddate))) == 3
@@ -199,3 +209,13 @@ def test_ftp():
     s = Scraper(pattern)
     timerange = TimeRange('2016/5/18 15:28:00', '2016/5/20 16:30:50')
     assert len(s.filelist(timerange)) == 2
+
+
+@pytest.mark.remote_data
+def test_filelist_url_missing_directory():
+    # Asserts solution to ticket #2684.
+    # Attempting to access data for the year 1960 results in a 404, so no files are returned.
+    pattern = 'http://lasp.colorado.edu/eve/data_access/evewebdataproducts/level2/%Y/%j/'
+    s = Scraper(pattern)
+    timerange = TimeRange('1960/01/01 00:00:00', '1960/01/02 00:00:00')
+    assert len(s.filelist(timerange)) == 0