Repository: obspy/obspy

Issue:
MassDownloader sometimes fails with an error of IncompleteRead
#### Description
We encountered an error when using MassDownloader to download a large number of waveforms. The tricky part of the problem is that it occurs pretty randomly and rarely, which makes me believe this is an issue related to the network rather than the code itself. Typically the error message looks like this when it fails:
```
...
[2020-04-19 12:25:23,977] - obspy.clients.fdsn.mass_downloader - INFO: Client 'IRIS' - Successfully downloaded 27 channels (of 27)
[2020-04-19 12:25:25,226] - obspy.clients.fdsn.mass_downloader - INFO: Client 'IRIS' - Successfully downloaded 22 channels (of 25)
[2020-04-19 12:25:32,274] - obspy.clients.fdsn.mass_downloader - INFO: Client 'IRIS' - Successfully downloaded 19 channels (of 23)
[2020-04-19 12:25:39,047] - obspy.clients.fdsn.mass_downloader - INFO: Client 'IRIS' - Successfully downloaded 20 channels (of 23)
Traceback (most recent call last):
  File "/usr/lib/python3.6/http/client.py", line 556, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/lib/python3.6/http/client.py", line 523, in _read_next_chunk_size
    return int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/http/client.py", line 573, in _readall_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/lib/python3.6/http/client.py", line 558, in _get_chunk_left
    raise IncompleteRead(b'')
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download2006.py", line 24, in <module>
    download_events(cat,yeartag=str(yr))
  File "/home/usarray/rfteledownload.py", line 83, in download_events
    stationxml_storage=site_storage)
  File "/home/.local/lib/python3.6/site-packages/obspy/clients/fdsn/mass_downloader/mass_downloader.py", line 207, in download
    threads_per_client=threads_per_client)
  File "/home/.local/lib/python3.6/site-packages/obspy/clients/fdsn/mass_downloader/download_helpers.py", line 880, in download_mseed
    [(self.client, self.client_name, chunk) for chunk in chunks])
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/.local/lib/python3.6/site-packages/obspy/clients/fdsn/mass_downloader/download_helpers.py", line 865, in star_download_mseed
    *args, logger=self.logger)
  File "/home/.local/lib/python3.6/site-packages/obspy/clients/fdsn/mass_downloader/utils.py", line 242, in download_and_split_mseed_bulk
    client.get_waveforms_bulk(bulk, filename=temp_filename)
  File "/home/.local/lib/python3.6/site-packages/obspy/clients/fdsn/client.py", line 1024, in get_waveforms_bulk
    data=bulk)
  File "/home/.local/lib/python3.6/site-packages/obspy/clients/fdsn/client.py", line 1396, in _download
    timeout=self.timeout, use_gzip=use_gzip)
  File "/home/.local/lib/python3.6/site-packages/obspy/clients/fdsn/client.py", line 1817, in download_url
    data = io.BytesIO(f.read())
  File "/usr/lib/python3.6/http/client.py", line 466, in read
    return self._readall_chunked()
  File "/usr/lib/python3.6/http/client.py", line 580, in _readall_chunked
    raise IncompleteRead(b''.join(value))
http.client.IncompleteRead: IncompleteRead(1970176 bytes read)
``` 
There are three exceptions thrown, but if we look at the lines in `/usr/lib/python3.6/http/client.py` referenced, one shouldn't be triggered while handling the other (as the error message says). It is more like three independent threads all got terminated at the same time. So the real problem here is that some exceptions in `/usr/lib/python3.6/http/client.py` is not captured and handled properly by `obspy/clients/fdsn/client.py`.

Note that this is on ObsPy 1.2.1.

#### Potential Fix
I was able to fix it by adding exception handler to https://github.com/obspy/obspy/blob/e4aeb46ceb7ef92819fe887288724583579e1f23/obspy/clients/fdsn/client.py#L1383-L1389 like this:
```python
    def _download(self, url, return_string=False, data=None, use_gzip=True):
        try:
            code, data = download_url(
                url, opener=self._url_opener, headers=self.request_headers,
                debug=self.debug, return_string=return_string, data=data,
                timeout=self.timeout, use_gzip=use_gzip)
        except Exception as e:
            raise FDSNException("Unknown Error: %s" % str(e))
        raise_on_error(code, data)
        return data
```
The problem is that the `raise_on_error(code, data)` function assumes the `download_url(…)` function will always succeed and return either the data or some error code. However, if we go into that function, we see that line of [`data = io.BytesIO(f.read())`](https://github.com/obspy/obspy/blob/e4aeb46ceb7ef92819fe887288724583579e1f23/obspy/clients/fdsn/client.py#L1806) where our code fails is outside of any try blocks. By adding the new exception, the output with network interruption will look like this:
```
...
[2020-04-30 12:04:36,021] - obspy.clients.fdsn.mass_downloader - INFO: Client 'NCEDC' - Successfully downloaded 3 channels (of 18)
[2020-04-30 12:04:37,716] - obspy.clients.fdsn.mass_downloader - INFO: Client 'NCEDC' - Successfully downloaded 18 channels (of 18)
[2020-04-30 12:04:38,647] - obspy.clients.fdsn.mass_downloader - ERROR: Client 'NCEDC' - Unknown Error: IncompleteRead(1191936 bytes read)
[2020-04-30 12:04:38,685] - obspy.clients.fdsn.mass_downloader - ERROR: Client 'NCEDC' - Unknown Error (URLError): <urlopen error [Errno -3] Temporary failure in name resolution>
...
```
Note that we now see the `IncompleteRead` in the error log, and the code will skip the data and keep running (which may not be good because the data set will be incomplete, but still better than failing).

#### To Reproduce
The trick I used to reproduce the error is to just disconnect my network when I see the log indicating it’s downloading waveform, by doing that, there is a pretty good chance (still not 100%) to see that `IncompleteRead` exception. 

#### Files
I do think this is a generic issue so it should be able to be reproduced downloading any data. Anyhow, I attach below the code we use as a reference. Note that the author is @pavlis.

filename: `download.py`
```python
import time
from rfteledownload import read_centers
from rfteledownload import get_catalog
from rfteledownload import download_events

centers=read_centers()
yr=2006
t0=time.time()
print("Running get_catalog")
cat=get_catalog(centers,year=yr)
fname=("events_%d.xml" % yr)
cat.write(fname,format='QUAKEML')
t1=time.time()
print("Elapsed time for to fetch catalog=",t1-t0)
print("Running download_events")
t1=time.time()
download_events(cat,yeartag=str(yr))
t2=time.time()
print("Elapsed time to download events=",t2-t1)
``` 

filename: `rfteledownload.py`
```python
import time
from obspy import UTCDateTime
from obspy.clients.fdsn.mass_downloader import RectangularDomain, \
    Restrictions, MassDownloader
from obspy.clients.fdsn import Client


def read_centers(file='year_centers.txt'):
    centers={}
    # More elegant method is to use with, but not necessary
    # for this private program.
    fh=open(file,mode='r')
    for line in fh:
        words=line.split()
        yr=int(words[0])
        lon=words[1]
        lat=words[2]
        centers[yr]=[lon,lat]
    return centers

def get_catalog(centers, year=2005, minmag=4.9):
    client=Client("IRIS")
    tstart=UTCDateTime(year,1,1,0,0)
    tend=UTCDateTime(year+1,1,1,0,0)
    coords=centers[year]
    lon0=coords[0]
    lat0=coords[1]
    cat=client.get_events(starttime=tstart,endtime=tend,
                          latitude=lat0,longitude=lon0,
                          minradius=25.0,maxradius=95.0,
                          minmagnitude=minmag)
    return cat

def download_events(cat,yeartag="2005",firstid=1):
    wf_storage=yeartag + "/{starttime}/{network}_{station}_{location}.mseed"
    site_storage="site_"+yeartag+".dir"
    mdl = MassDownloader()
    domain = RectangularDomain(minlatitude=20.0,maxlatitude=54.0,
        minlongitude=-135, maxlongitude=-55)
    count_evid=firstid
    for event in cat:
        t0=time.time()
        o=event.preferred_origin()
        # Time attribute is already a UTCDateTime object
        origin_time=o.time
        restrictions = Restrictions (
         starttime=origin_time,
         endtime=origin_time + 3600.0,
         reject_channels_with_gaps=True,
         minimum_length=0.95,
         minimum_interstation_distance_in_m=100.0,
         channel_priorities=["HH[ZNE12]","BH[ZNE12]"],
         location_priorities=["","00","10"])
        
        wf_storage=("%s/event%d" % (yeartag,count_evid))
        mdl.download(domain,restrictions,
            mseed_storage=wf_storage,
            stationxml_storage=site_storage)
        dt=time.time()-t0
        print("Event ",count_evid," download time (s)=",dt)
        count_evid += 1
```

filename: `year_centers.txt`
```
2005 -122.0 41.0
2006 -119.0 41.0
```





Discussion:
I'd also guess that this is a network connection issue. 

The mass downloader already catches a bunch of different errors, but this particular one not. To fix it, just add the `http.client.IncompleteRead` to the list of errors that are caught: https://github.com/obspy/obspy/blob/88687a146a7c3ca8f35c608db2243cf5fed6813c/obspy/clients/fdsn/mass_downloader/utils.py#L32

Please also consider opening a pull request so others can benefit from your changes.

Resolving patch:
diff --git a/obspy/clients/fdsn/mass_downloader/utils.py b/obspy/clients/fdsn/mass_downloader/utils.py
--- a/obspy/clients/fdsn/mass_downloader/utils.py
+++ b/obspy/clients/fdsn/mass_downloader/utils.py
@@ -25,8 +25,10 @@
 
 if sys.version_info.major == 2:
     from urllib2 import HTTPError, URLError
+    from httplib import HTTPException
 else:
     from urllib.error import HTTPError, URLError
+    from http.client import HTTPException
 
 import obspy
 from obspy.core import compatibility
@@ -37,7 +39,7 @@
 
 # Different types of errors that can happen when downloading data via the
 # FDSN clients.
-ERRORS = [FDSNException, HTTPError, URLError, socket_timeout]
+ERRORS = [FDSNException, HTTPException, HTTPError, URLError, socket_timeout]
 
 # Python 2 does have special classes for connection errors.
 if sys.version_info.major == 2:


Test patch:
diff --git a/obspy/clients/fdsn/tests/test_mass_downloader.py b/obspy/clients/fdsn/tests/test_mass_downloader.py
--- a/obspy/clients/fdsn/tests/test_mass_downloader.py
+++ b/obspy/clients/fdsn/tests/test_mass_downloader.py
@@ -20,9 +20,15 @@
 import os
 import shutil
 from socket import timeout as socket_timeout
+import sys
 import tempfile
 import unittest
 
+if sys.version_info.major == 2:
+    from httplib import HTTPException
+else:
+    from http.client import HTTPException
+
 import numpy as np
 
 import obspy
@@ -2163,6 +2169,21 @@ def test_download_mseed(self, patch_check_data, patch_download_mseed):
         # is just an info message.
         self.assertEqual(c.logger.error.call_count, 0)
 
+        patch_check_data.reset_mock()
+        patch_download_mseed.reset_mock()
+        c = self._init_client()
+        c.stations = {
+            ("A", "A"): Station("A", "A", 0, 10, copy.deepcopy(channels))
+        }
+
+        patch_download_mseed.side_effect = HTTPException("disconnected")
+
+        c.download_mseed()
+        self.assertEqual(patch_check_data.call_count, 1)
+        self.assertEqual(patch_download_mseed.call_count, 1)
+        # The error logger should have been called once
+        self.assertEqual(c.logger.error.call_count, 1)
+
     @mock.patch("obspy.clients.fdsn.mass_downloader."
                 "utils.download_stationxml")
     @mock.patch("obspy.clients.fdsn.mass_downloader."