Repository: obspy/obspy

Issue:
Minor correction in obspy.signal.PPSD
Hi, was trying to do very precise PSD processing of our archive. Unfortunately I need to work with exact samples and expect 48 segments (30 min. each) per day. The PPSD calculation always uses a single sample per segment more than it should and results in to the final daily segment being dropped.

Here is some demo code:

    import obspy
    import numpy as np

    header = {
        "starttime":  obspy.UTCDateTime("2017-01-01T00:00:00"),
        "network": "GB",
        "station": "MCH1",
        "channel": "BHE"
    }

    # 24.5 hours to account for 50% overlap through the next day
    print tr = obspy.Trace(data=np.arange(30 * 60 * 49, dtype=np.int32), header=header)
    >>> GB.MCH1..BHE | 2017-01-01T00:00:00.000000Z - 2017-01-02T00:29:59.000000Z | 1.0 Hz, 88200 samples

    ppsd = obspy.signal.PPSD(
        tr.stats,
        obspy.read_inventory("http://www.orfeus-eu.org/fdsnws/station/1/query?sta=MCH1&cha=BHE&level=response"),
    )

    ppsd.add(tr)

    print len(ppsd._times_processed)
    >>> 47

When we add a single second extra to the trace it will give 48 PSDs as expected but that is incorrect.

Here is a list of the segments it calculates:

    GB.MCH1..BHE | 2017-01-01T00:00:00.000000Z - 2017-01-01T01:00:00.000000Z | 1.0 Hz, 3601 samples
    GB.MCH1..BHE | 2017-01-01T00:30:00.000000Z - 2017-01-01T01:30:00.000000Z | 1.0 Hz, 3601 samples
    ...
    GB.MCH1..BHE | 2017-01-01T22:30:00.000000Z - 2017-01-01T23:30:00.000000Z | 1.0 Hz, 3601 samples
    GB.MCH1..BHE | 2017-01-01T23:00:00.000000Z - 2017-01-02T00:00:00.000000Z | 1.0 Hz, 3601 samples

It shows that the final sample used per segment overlaps and we are using 3601 samples per segment, not 3600 as we should. IMHO It can be fixed easily by subtracting tr.stats.delta from the end time of each segment.

I think it could be changed, what do you reckon @megies?
  


Discussion:
I'm aware of limitations of the current approach, feeding in Traces. For cases that need to be ultra-exact, probably it would be easiest to add a new, additional method to add individual data slices (and raising an exception on wrong-length user input)? That would then leave the preparing the slices to the user and thus be fully transparent to them.

About the additional sample (making it 3601 in this case), I agree that this should be fixed..

(So this issue probably mixes two things that should be addressed, one for the maintenance line and one for master?)
  
> (So this issue probably mixes two things that should be addressed, one for the maintenance line and one for master?)

I was only referencing the last sample problem, but I'm writing some code atm to address this.
  

Resolving patch:
diff --git a/obspy/signal/spectral_estimation.py b/obspy/signal/spectral_estimation.py
--- a/obspy/signal/spectral_estimation.py
+++ b/obspy/signal/spectral_estimation.py
@@ -718,7 +718,7 @@ def add(self, stream, verbose=False):
                 continue
             t1 = tr.stats.starttime
             t2 = tr.stats.endtime
-            while t1 + self.ppsd_length <= t2:
+            while t1 + self.ppsd_length - tr.stats.delta <= t2:
                 if self.__check_time_present(t1):
                     msg = "Already covered time spans detected (e.g. %s), " + \
                           "skipping these slices."
@@ -727,7 +727,8 @@ def add(self, stream, verbose=False):
                 else:
                     # throw warnings if trace length is different
                     # than ppsd_length..!?!
-                    slice = tr.slice(t1, t1 + self.ppsd_length)
+                    slice = tr.slice(t1, t1 + self.ppsd_length
+                                     - tr.stats.delta)
                     # XXX not good, should be working in place somehow
                     # XXX how to do it with the padding, though?
                     success = self.__process(slice)


Test patch:
diff --git a/obspy/signal/tests/test_spectral_estimation.py b/obspy/signal/tests/test_spectral_estimation.py
--- a/obspy/signal/tests/test_spectral_estimation.py
+++ b/obspy/signal/tests/test_spectral_estimation.py
@@ -677,6 +677,32 @@ def test_ppsd_temporal_plot(self):
             ppsd.plot_temporal([0.1, 1, 10], filename=ic.name, show=False,
                                **restrictions)
 
+    def test_exclude_last_sample(self):
+
+        header = {
+            "starttime":  UTCDateTime("2017-01-01T00:00:00"),
+            "network": "GR",
+            "station": "FUR",
+            "channel": "BHZ"
+        }
+
+        # 49 segments of 30 minutes to allow 30 minutes overlap in next day
+        tr = Trace(data=np.arange(30 * 60 * 4, dtype=np.int32), header=header)
+
+        ppsd = PPSD(
+            tr.stats,
+            read_inventory()
+        )
+
+        ppsd.add(tr)
+
+        self.assertEqual(3, len(ppsd._times_processed))
+        self.assertEqual(3600, ppsd.len)
+
+        for i, time in enumerate(ppsd._times_processed):
+            current = UTCDateTime("2017-01-01T00:00:00") + (i * 30 * 60)
+            self.assertTrue(UTCDateTime(time) == current)
+
     def test_ppsd_spectrogram_plot(self):
         """
         Test spectrogram type plot of PPSD