Repository: obspy/obspy

Issue:
[MassDownloader] NotImplementedError (starttime > first["endtime"] or ...)
Here is another error I've got when using the mass downloader:

```
Traceback (most recent call last):
  File "./10_downloader.py", line 122, in <module>
    stationxml_storage=stationxml_storage)
  File "/home/legovini/obspy/obspy/clients/fdsn/mass_downloader/mass_downloader.py", line 201, in download
    threads_per_client=threads_per_client)
  File "/home/legovini/obspy/obspy/clients/fdsn/mass_downloader/download_helpers.py", line 857, in download_mseed
    [(self.client, self.client_name, chunk) for chunk in chunks])
  File "/home/legovini/miniconda3/envs/sismo/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/legovini/miniconda3/envs/sismo/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/home/legovini/miniconda3/envs/sismo/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/legovini/miniconda3/envs/sismo/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/legovini/obspy/obspy/clients/fdsn/mass_downloader/download_helpers.py", line 842, in star_download_mseed
    *args, logger=self.logger)
  File "/home/legovini/obspy/obspy/clients/fdsn/mass_downloader/utils.py", line 245, in download_and_split_mseed_bulk
    c=filenames[channel_id])
  File "/home/legovini/obspy/obspy/clients/fdsn/mass_downloader/utils.py", line 143, in get_filename
    raise NotImplementedError
NotImplementedError
```

I took a look at utils.py:143, the failing condition is:
```
# Make sure the assumptions about the type of overlap are correct.
if starttime > first["endtime"] or endtime < second["starttime"]:
    raise NotImplementedError
```

Couldn't be this just due to a corrupted file in the datacenter? Should ObsPy handle this situation more gracefully? I could try to make this reproducible, if you're interested in looking into it.


Discussion:
Can you post a minimal example reproducing this? I don't see right now how this can happen so I guess its some corrupt file? But would be nice to look at it.
could it be an un-sorted mseed file ?
@krischer Unfortunately now I forgot which station and data interval was causing this error. The next time I encounter the issue I'll try to isolate what is causing the problem and let you have a minimal example.
> could it be an un-sorted mseed file ?

I don't think so because the records are sorted by time right before this line is called.



> @krischer Unfortunately now I forgot which station and data interval was causing this error. The next time I encounter the issue I'll try to isolate what is causing the problem and let you have a minimal example.

Please do - I don't really understand how this can happen so I would need to be able to reproduce it to fix it. 

I'll remove this issue from the milestone until we have a reproducible example.
Cc: @petrrr 

OK, I'm back with an example to reproduce the issue. The following script is supposed to download everything available between 1985 and 1999 for a station named `TNV` that lies below -59S. By "everything" I mean the data for all the networks, channels, locations for `TNV` within the defined `RectangularDomain`. I don't care if there are gaps or other issues with the data, I want to retrieve *everything*. (No data actually exists before 1991.)

The process is split in yearly requests (the `for` loop) because sometimes data centers do not like extremely large requests (this is probably not the case, but I've made bigger ones). Suggestions on doing this in a better way are of course welcome. 

```
#!/usr/bin/env python3

import obspy
from obspy.clients.fdsn import mass_downloader as MD

domain = MD.RectangularDomain(minlatitude=-90, maxlatitude=-59, minlongitude=-180, maxlongitude=180)

restrictions = MD.Restrictions(
    # Get data for a whole year
    starttime=obspy.UTCDateTime(1985, 1, 1),
    endtime=obspy.UTCDateTime(1985, 12, 31),

    # We want daily chunks
    chunklength_in_sec=86400,

    station="TNV",
    channel_priorities=["*"],
    location_priorities=["*"],

    reject_channels_with_gaps=False,
    minimum_length=0.0,
    minimum_interstation_distance_in_m=0.0,
    sanitize=False
)

mdl = MD.MassDownloader(providers=['IRIS'])

# Years to download (inclusive)
first_year = 1985
last_year  = 1999

for y in range(first_year, last_year+1):
    print("### YEAR: " + str(y))
    restrictions.starttime=obspy.UTCDateTime(y, 1, 1)
    restrictions.endtime=obspy.UTCDateTime(y, 12, 31)

    mdl.download(domain, restrictions, download_chunk_size_in_mb=50,
            threads_per_client=3, mseed_storage="waveforms/" + str(y),
            stationxml_storage="stations/" + str(y))
    print("### END YEAR: " + str(y))
```

This script systematically fails in this way:
```
[2018-06-16 16:50:46,906] - obspy.clients.fdsn.mass_downloader - INFO: Client 'IRIS' - No data available for request.
[2018-06-16 16:50:47,382] - obspy.clients.fdsn.mass_downloader - INFO: Client 'IRIS' - No data available for request.
Traceback (most recent call last):
  File "./10_downloader.py", line 42, in <module>
    stationxml_storage="stations/" + str(y))
  File "/home/legovini/obspy/obspy/clients/fdsn/mass_downloader/mass_downloader.py", line 201, in download
    threads_per_client=threads_per_client)
  File "/home/legovini/obspy/obspy/clients/fdsn/mass_downloader/download_helpers.py", line 857, in download_mseed
    [(self.client, self.client_name, chunk) for chunk in chunks])
  File "/home/legovini/miniconda3/envs/sismo/lib/python3.6/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/legovini/miniconda3/envs/sismo/lib/python3.6/multiprocessing/pool.py", line 644, in get
    raise self._value
  File "/home/legovini/miniconda3/envs/sismo/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/legovini/miniconda3/envs/sismo/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/legovini/obspy/obspy/clients/fdsn/mass_downloader/download_helpers.py", line 842, in star_download_mseed
    *args, logger=self.logger)
  File "/home/legovini/obspy/obspy/clients/fdsn/mass_downloader/utils.py", line 245, in download_and_split_mseed_bulk
    c=filenames[channel_id])
  File "/home/legovini/obspy/obspy/clients/fdsn/mass_downloader/utils.py", line 143, in get_filename
    raise NotImplementedError
NotImplementedError
```

I did this using the tip of ObsPy's master branch, currently 06906f389bdaeb5fa4c4bd0cab9f066082e7da42. I hope this helps!
I stripped down the example, this won't require you to download >2GB of data to reproduce the issue:
```
#!/usr/bin/env python3

import obspy
from obspy.clients.fdsn import mass_downloader as MD

domain = MD.RectangularDomain(minlatitude=-90, maxlatitude=-66, minlongitude=-180, maxlongitude=180)

restrictions = MD.Restrictions(
    starttime=obspy.UTCDateTime(1991, 2, 21),
    endtime=obspy.UTCDateTime(1991, 2, 23),

    # We want daily chunks
    chunklength_in_sec=86400,

    station="TNV",
    channel="VHZ"
)

mdl = MD.MassDownloader(providers=['IRIS'])

mdl.download(domain, restrictions, mseed_storage="waveforms/", stationxml_storage="stations/")
```
I can systematically reproduce the issue, just remember to delete the `waveforms/` directory between the tries. I added a `print(candicates)` at `utils.py:157`:
```
# Make sure the assumptions about the type of overlap are correct.
if starttime > first["endtime"] or endtime < second["starttime"]:
    print(candidates)
    raise NotImplementedError
```
The result is:
```
[...]
[2018-06-17 18:02:32,835] - obspy.clients.fdsn.mass_downloader - INFO: Client 'IRIS' - Status for 2 time intervals/channels before downloading: NEEDS_DOWNLOADING
[{'starttime': UTCDateTime(1991, 2, 21, 0, 0), 'endtime': UTCDateTime(1991, 2, 22, 0, 0), 'filename': 'waveforms/MN.TNV..VHZ__19910221T000000Z__19910222T000000Z.mseed', 'curr
ent_latest_endtime': UTCDateTime(1991, 2, 21, 21, 13, 50, 430000), 'sequence_number': 5}, {'starttime': UTCDateTime(1991, 2, 22, 0, 0), 'endtime': UTCDateTime(1991, 2, 23, 0,
 0), 'filename': 'waveforms/MN.TNV..VHZ__19910222T000000Z__19910223T000000Z.mseed', 'current_latest_endtime': None, 'sequence_number': None}]
Traceback (most recent call last):
[...]
```

This is probably important: if I comment out the `chunklength_in_sec=86400` line the script works fine. 

Resolving patch:
diff --git a/obspy/io/mseed/scripts/recordanalyzer.py b/obspy/io/mseed/scripts/recordanalyzer.py
--- a/obspy/io/mseed/scripts/recordanalyzer.py
+++ b/obspy/io/mseed/scripts/recordanalyzer.py
@@ -165,7 +165,7 @@ def _read_fixed_header(self):
         # Read and unpack.
         self.file.seek(self.record_offset, 0)
         fixed_header = self.file.read(48)
-        encoding = native_str('%s20c2H3Bx4H4Bl2H' % self.endian)
+        encoding = native_str('%s20c2H3Bx2H2h4Bl2h' % self.endian)
         try:
             header_item = unpack(encoding, fixed_header)
         except Exception:
@@ -343,7 +343,11 @@ def __str__(self):
                    endian)
         ret_val += 'FIXED SECTION OF DATA HEADER\n'
         for key in self.fixed_header.keys():
-            ret_val += '\t%s: %s\n' % (key, self.fixed_header[key])
+            # Don't print empty values to ease testing.
+            if self.fixed_header[key] != "":
+                ret_val += '\t%s: %s\n' % (key, self.fixed_header[key])
+            else:
+                ret_val += '\t%s:\n' % (key)
         ret_val += '\nBLOCKETTES\n'
         for key in self.blockettes.keys():
             ret_val += '\t%i:' % key
diff --git a/obspy/io/mseed/util.py b/obspy/io/mseed/util.py
--- a/obspy/io/mseed/util.py
+++ b/obspy/io/mseed/util.py
@@ -795,7 +795,7 @@ def _parse_time(values):
         elif (samp_rate_factor < 0) and (samp_rate_mult) > 0:
             samp_rate = -1.0 * float(samp_rate_mult) / float(samp_rate_factor)
         elif (samp_rate_factor < 0) and (samp_rate_mult) < 0:
-            samp_rate = -1.0 / float(samp_rate_factor * samp_rate_mult)
+            samp_rate = 1.0 / float(samp_rate_factor * samp_rate_mult)
         else:
             samp_rate = 0
 


Test patch:
diff --git a/obspy/io/mseed/tests/data/single_record_negative_sr_fact_and_mult.mseed b/obspy/io/mseed/tests/data/single_record_negative_sr_fact_and_mult.mseed
new file mode 100644
Binary files /dev/null and b/obspy/io/mseed/tests/data/single_record_negative_sr_fact_and_mult.mseed differ
diff --git a/obspy/io/mseed/tests/test_mseed_util.py b/obspy/io/mseed/tests/test_mseed_util.py
--- a/obspy/io/mseed/tests/test_mseed_util.py
+++ b/obspy/io/mseed/tests/test_mseed_util.py
@@ -92,6 +92,7 @@ class MSEEDUtilTestCase(unittest.TestCase):
     Tests suite for util module of obspy.io.mseed.
     """
     def setUp(self):
+        self.maxDiff = None
         # Directory where the test files are located
         self.path = os.path.dirname(__file__)
         # mseed steim compression is big endian
@@ -201,6 +202,34 @@ def test_get_record_information(self):
         self.assertEqual(info['number_of_records'], 2)
         self.assertEqual(info['excess_bytes'], 0)
 
+    def test_get_record_information_negative_sr_rate_and_mult(self):
+        """
+        Tests the method for negative sampling rate factors and multipliers.
+        """
+        filename = os.path.join(
+            self.path, 'data', 'single_record_negative_sr_fact_and_mult.mseed')
+        info = util.get_record_information(filename)
+        self.assertEqual(info, {
+            'activity_flags': 0,
+            'byteorder': '>',
+            'channel': 'VHZ',
+            'data_quality_flags': 0,
+            'encoding': 10,
+            'endtime': UTCDateTime(1991, 2, 21, 23, 59, 50, 430000),
+            'excess_bytes': 0,
+            'filesize': 4096,
+            'io_and_clock_flags': 0,
+            'location': '',
+            'network': 'MN',
+            'npts': 60,
+            'number_of_records': 1,
+            'record_length': 4096,
+            'samp_rate': 0.1,
+            'starttime': UTCDateTime(1991, 2, 21, 23, 50, 0, 430000),
+            'station': 'TNV',
+            'time_correction': 0
+        })
+
     def test_issue2069(self):
         """
         Tests the util._get_ms_file_info method with sample rate of 0.
diff --git a/obspy/io/mseed/tests/test_recordanalyzer.py b/obspy/io/mseed/tests/test_recordanalyzer.py
--- a/obspy/io/mseed/tests/test_recordanalyzer.py
+++ b/obspy/io/mseed/tests/test_recordanalyzer.py
@@ -13,6 +13,7 @@
 
 class RecordAnalyserTestCase(unittest.TestCase):
     def setUp(self):
+        self.maxDiff = None
         self.test_file = os.path.join(os.path.dirname(__file__),
                                       'data',
                                       'timingquality.mseed')
@@ -27,37 +28,37 @@ def test_default_record(self):
 Header Endianness: Big Endian
 
 FIXED SECTION OF DATA HEADER
-	Sequence number: 763445
-	Data header/quality indicator: D
-	Station identifier code: BGLD
-	Location identifier: 
-	Channel identifier: EHE
-	Network code: BW
-	Record start time: 2007-12-31T23:59:59.915000Z
-	Number of samples: 412
-	Sample rate factor: 200
-	Sample rate multiplier: 1
-	Activity flags: 0
-	I/O and clock flags: 0
-	Data quality flags: 0
-	Number of blockettes that follow: 2
-	Time correction: -1500
-	Beginning of data: 64
-	First blockette: 48
+    Sequence number: 763445
+    Data header/quality indicator: D
+    Station identifier code: BGLD
+    Location identifier:
+    Channel identifier: EHE
+    Network code: BW
+    Record start time: 2007-12-31T23:59:59.915000Z
+    Number of samples: 412
+    Sample rate factor: 200
+    Sample rate multiplier: 1
+    Activity flags: 0
+    I/O and clock flags: 0
+    Data quality flags: 0
+    Number of blockettes that follow: 2
+    Time correction: -1500
+    Beginning of data: 64
+    First blockette: 48
 
 BLOCKETTES
-	1000:	Encoding Format: 10
-		Word Order: 1
-		Data Record Length: 9
-	1001:	Timing quality: 55
-		mu_sec: 0
-		Frame count: -73
+    1000:    Encoding Format: 10
+        Word Order: 1
+        Data Record Length: 9
+    1001:    Timing quality: 55
+        mu_sec: 0
+        Frame count: -73
 
 CALCULATED VALUES
-	Corrected Starttime: 2007-12-31T23:59:59.765000Z
+    Corrected Starttime: 2007-12-31T23:59:59.765000Z
 
 ''' % (self.test_file,)  # noqa
-        self.assertEqual(expected, out.stdout)  # noqa
+        self.assertEqual(expected, out.stdout.replace("\t", "    "))  # noqa
 
     def test_second_record(self):
         with CatchOutput() as out:
@@ -69,37 +70,37 @@ def test_second_record(self):
 Header Endianness: Big Endian
 
 FIXED SECTION OF DATA HEADER
-	Sequence number: 763446
-	Data header/quality indicator: D
-	Station identifier code: BGLD
-	Location identifier: 
-	Channel identifier: EHE
-	Network code: BW
-	Record start time: 2008-01-01T00:00:01.975000Z
-	Number of samples: 412
-	Sample rate factor: 200
-	Sample rate multiplier: 1
-	Activity flags: 0
-	I/O and clock flags: 0
-	Data quality flags: 0
-	Number of blockettes that follow: 2
-	Time correction: -1500
-	Beginning of data: 64
-	First blockette: 48
+    Sequence number: 763446
+    Data header/quality indicator: D
+    Station identifier code: BGLD
+    Location identifier:
+    Channel identifier: EHE
+    Network code: BW
+    Record start time: 2008-01-01T00:00:01.975000Z
+    Number of samples: 412
+    Sample rate factor: 200
+    Sample rate multiplier: 1
+    Activity flags: 0
+    I/O and clock flags: 0
+    Data quality flags: 0
+    Number of blockettes that follow: 2
+    Time correction: -1500
+    Beginning of data: 64
+    First blockette: 48
 
 BLOCKETTES
-	1000:	Encoding Format: 10
-		Word Order: 1
-		Data Record Length: 9
-	1001:	Timing quality: 70
-		mu_sec: 0
-		Frame count: -73
+    1000:    Encoding Format: 10
+        Word Order: 1
+        Data Record Length: 9
+    1001:    Timing quality: 70
+        mu_sec: 0
+        Frame count: -73
 
 CALCULATED VALUES
-	Corrected Starttime: 2008-01-01T00:00:01.825000Z
+    Corrected Starttime: 2008-01-01T00:00:01.825000Z
 
 ''' % (self.test_file,)  # noqa
-        self.assertEqual(expected, out.stdout)  # noqa
+        self.assertEqual(expected, out.stdout.replace("\t", "    "))  # noqa
 
     def test_record_with_data_offset_zero(self):
         """
@@ -118,35 +119,35 @@ def test_record_with_data_offset_zero(self):
 Header Endianness: Big Endian
 
 FIXED SECTION OF DATA HEADER
-	Sequence number: 1
-	Data header/quality indicator: D
-	Station identifier code: PANIX
-	Location identifier: 
-	Channel identifier: LHZ
-	Network code: CH
-	Record start time: 2016-08-21T01:43:37.000000Z
-	Number of samples: 0
-	Sample rate factor: 1
-	Sample rate multiplier: 1
-	Activity flags: 4
-	I/O and clock flags: 0
-	Data quality flags: 0
-	Number of blockettes that follow: 2
-	Time correction: 0
-	Beginning of data: 0
-	First blockette: 48
+    Sequence number: 1
+    Data header/quality indicator: D
+    Station identifier code: PANIX
+    Location identifier:
+    Channel identifier: LHZ
+    Network code: CH
+    Record start time: 2016-08-21T01:43:37.000000Z
+    Number of samples: 0
+    Sample rate factor: 1
+    Sample rate multiplier: 1
+    Activity flags: 4
+    I/O and clock flags: 0
+    Data quality flags: 0
+    Number of blockettes that follow: 2
+    Time correction: 0
+    Beginning of data: 0
+    First blockette: 48
 
 BLOCKETTES
-	1000:	Encoding Format: 11
-		Word Order: 1
-		Data Record Length: 9
-	201:	NOT YET IMPLEMENTED
+    1000:    Encoding Format: 11
+        Word Order: 1
+        Data Record Length: 9
+    201:    NOT YET IMPLEMENTED
 
 CALCULATED VALUES
-	Corrected Starttime: 2016-08-21T01:43:37.000000Z
+    Corrected Starttime: 2016-08-21T01:43:37.000000Z
 
 ''' % (filename,)  # noqa
-        self.assertEqual(expected, out.stdout)  # noqa
+        self.assertEqual(expected, out.stdout.replace("\t", "    "))  # noqa
 
         with CatchOutput() as out:
             obspy_recordanalyzer(['-n', '2', filename])
@@ -157,37 +158,84 @@ def test_record_with_data_offset_zero(self):
 Header Endianness: Big Endian
 
 FIXED SECTION OF DATA HEADER
-	Sequence number: 189
-	Data header/quality indicator: D
-	Station identifier code: PANIX
-	Location identifier: 
-	Channel identifier: LHZ
-	Network code: CH
-	Record start time: 2016-08-21T01:45:31.000000Z
-	Number of samples: 262
-	Sample rate factor: 1
-	Sample rate multiplier: 1
-	Activity flags: 68
-	I/O and clock flags: 32
-	Data quality flags: 0
-	Number of blockettes that follow: 2
-	Time correction: 0
-	Beginning of data: 64
-	First blockette: 48
+    Sequence number: 189
+    Data header/quality indicator: D
+    Station identifier code: PANIX
+    Location identifier:
+    Channel identifier: LHZ
+    Network code: CH
+    Record start time: 2016-08-21T01:45:31.000000Z
+    Number of samples: 262
+    Sample rate factor: 1
+    Sample rate multiplier: 1
+    Activity flags: 68
+    I/O and clock flags: 32
+    Data quality flags: 0
+    Number of blockettes that follow: 2
+    Time correction: 0
+    Beginning of data: 64
+    First blockette: 48
 
 BLOCKETTES
-	1000:	Encoding Format: 11
-		Word Order: 1
-		Data Record Length: 9
-	1001:	Timing quality: 100
-		mu_sec: 0
-		Frame count: 7
+    1000:    Encoding Format: 11
+        Word Order: 1
+        Data Record Length: 9
+    1001:    Timing quality: 100
+        mu_sec: 0
+        Frame count: 7
 
 CALCULATED VALUES
-	Corrected Starttime: 2016-08-21T01:45:31.000000Z
+    Corrected Starttime: 2016-08-21T01:45:31.000000Z
 
 ''' % (filename,)  # noqa
-        self.assertEqual(expected, out.stdout)  # noqa
+        self.assertEqual(expected, out.stdout.replace("\t", "    "))  # noqa
+
+    def test_record_with_negative_sr_fact_and_mult(self):
+        """
+        Regression tests as there was an issue with the record analyzer for
+        negative sampling rates and factors.
+        """
+        filename = os.path.join(
+            os.path.dirname(__file__), 'data',
+            'single_record_negative_sr_fact_and_mult.mseed')
+
+        with CatchOutput() as out:
+            obspy_recordanalyzer([filename])
+
+        expected = '''FILE: %s
+Record Number: 0
+Record Offset: 0 byte
+Header Endianness: Big Endian
+
+FIXED SECTION OF DATA HEADER
+    Sequence number: 4
+    Data header/quality indicator: M
+    Station identifier code: TNV
+    Location identifier:
+    Channel identifier: VHZ
+    Network code: MN
+    Record start time: 1991-02-21T23:50:00.430000Z
+    Number of samples: 60
+    Sample rate factor: -10
+    Sample rate multiplier: -1
+    Activity flags: 0
+    I/O and clock flags: 0
+    Data quality flags: 0
+    Number of blockettes that follow: 1
+    Time correction: 0
+    Beginning of data: 64
+    First blockette: 48
+
+BLOCKETTES
+    1000:    Encoding Format: 10
+        Word Order: 1
+        Data Record Length: 12
+
+CALCULATED VALUES
+    Corrected Starttime: 1991-02-21T23:50:00.430000Z
+
+''' % (filename,)  # noqa
+        self.assertEqual(expected, out.stdout.replace("\t", "    "))  # noqa
 
 
 def suite():