Repository: gwastro/pycbc

Issue:
Sporadic error in SNR series generation in PyCBC Live
Reported by @ahnitz:
```
2018-07-07 10:26:10,410 atlas4 done clustering coinc triggers: 1478 triggers remaining
2018-07-07 10:26:10,471 atlas4 Generating SPAtmplt, 80.0s, 341678, starting from 26.9 Hz
Traceback (most recent call last):
File "/home/pycbc.live/projects/pycbc_live_production/O3exp/env/bin/pycbc_live", line 4, in <module>
__import__('pkg_resources').run_script('PyCBC===48da87', 'pycbc_live')
File "/home/pycbc.live/projects/pycbc_live_production/O3exp/env/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 654, in run_script
self.require(requires)[0].run_script(script_name, ns)
File "/home/pycbc.live/projects/pycbc_live_production/O3exp/env/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1434, in run_script
exec(code, namespace, namespace)
File "/home/pycbc.live/projects/pycbc_live_production/O3exp/env/lib/python2.7/site-packages/PyCBC-48da87-py2.7-linux-x86_64.egg/EGG-INFO/scripts/pycbc_live", line 515, in <module>
followup_ifos=usable_followup_ifos)
File "/home/pycbc.live/projects/pycbc_live_production/O3exp/env/lib/python2.7/site-packages/PyCBC-48da87-py2.7-linux-x86_64.egg/EGG-INFO/scripts/pycbc_live", line 114, in check_coincs
bank, followup_ifos)
File "/home/pycbc.live/projects/pycbc_live_production/O3exp/env/lib/python2.7/site-packages/PyCBC-48da87-py2.7-linux-x86_64.egg/EGG-INFO/scripts/pycbc_live", line 103, in compute_followup_data
check_state=(ifo in followup_ifos))
File "/home/pycbc.live/projects/pycbc_live_production/O3exp/env/lib/python2.7/site-packages/PyCBC-48da87-py2.7-linux-x86_64.egg/pycbc/filter/matchedfilter.py", line 1782, in compute_followup_snr_series
return snr[onsource_slice] * norm, stilde.psd
File "<decorator-gen-120>", line 2, in __getitem__
File "/home/pycbc.live/projects/pycbc_live_production/O3exp/env/lib/python2.7/site-packages/PyCBC-48da87-py2.7-linux-x86_64.egg/pycbc/types/array.py", line 65, in _convert
return fn(self, *args)
File "/home/pycbc.live/projects/pycbc_live_production/O3exp/env/lib/python2.7/site-packages/PyCBC-48da87-py2.7-linux-x86_64.egg/pycbc/types/array.py", line 738, in __getitem__
return self._getslice(index)
File "/home/pycbc.live/projects/pycbc_live_production/O3exp/env/lib/python2.7/site-packages/PyCBC-48da87-py2.7-linux-x86_64.egg/pycbc/types/timeseries.py", line 105, in _getslice
raise ValueError('Negative start index not supported')
ValueError: Negative start index not supported
```


Discussion:
I did a few artificial tests, trying to call `compute_followup_snr_series()` with a mock StrainBuffer. I can reproduce this error by forcing the trigger time to be before the `start_time` of the StrainBuffer (even just by one sample), but I doubt that is what caused the original error in `pycbc_live`. Nevertheless I still suspect this is an off-by-one issue somewhere in `compute_followup_snr_series()`. I am now running a full simulation of `pycbc_live` with tightly packed injections and hopefully the error will pop up again.
I managed to reproduce this with simulated strain and injections, indeed it happens when the trigger time is very close to the start of the block:
```
2018-07-17 08:09:30,062 node578 Generating SEOBNRv4_ROM, 16.0s, 1590, starting from 26.6 Hz
2018-07-17 08:09:30,084 node578 trig_time - snr.start_time = 0.047852
2018-07-17 08:09:30,105 node578 trig_time - snr.start_time = 0.038574
Traceback (most recent call last):
  File "/home/tito/sw_install/pycbc//bin/pycbc_live", line 4, in <module>
    __import__('pkg_resources').run_script('PyCBC===073fb5', 'pycbc_live')
  File "/home/tito/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 739, in run_script
    self.require(requires)[0].run_script(script_name, ns)
  File "/home/tito/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1500, in run_script
    exec(code, namespace, namespace)
  File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/EGG-INFO/scripts/pycbc_live", line 512, in <module>
    followup_ifos=usable_followup_ifos)
  File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/EGG-INFO/scripts/pycbc_live", line 114, in check_coincs
    bank, followup_ifos)
  File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/EGG-INFO/scripts/pycbc_live", line 103, in compute_followup_data
    check_state=(ifo in followup_ifos))
  File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/pycbc/filter/matchedfilter.py", line 1783, in compute_followup_snr_series
    return snr[onsource_slice] * norm, stilde.psd
  File "<decorator-gen-120>", line 2, in __getitem__
  File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/pycbc/types/array.py", line 65, in _convert
    return fn(self, *args)
  File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/pycbc/types/array.py", line 738, in __getitem__
    return self._getslice(index)
  File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/pycbc/types/timeseries.py", line 105, in _getslice
    raise ValueError('Negative start index not supported')
ValueError: Negative start index not supported
```
`~tito/projects/pycbc/debug_2249` @ CIT.
Tito, is this happening because we slice to the valid region first? If it
is so close to the boundary, maybe we should allow it to look further back
in the SNR time series anyway, it's not as if there isn't enough padding to
go O(50) ms.

On Tue, Jul 17, 2018 at 11:12 AM, Tito Dal Canton <notifications@github.com>
wrote:

> I managed to reproduce this with simulated strain and injections, indeed
> it happens when the trigger time is very close to the start of the block:
>
> 2018-07-17 08:09:30,062 node578 Generating SEOBNRv4_ROM, 16.0s, 1590, starting from 26.6 Hz
> 2018-07-17 08:09:30,084 node578 trig_time - snr.start_time = 0.047852
> 2018-07-17 08:09:30,105 node578 trig_time - snr.start_time = 0.038574
> Traceback (most recent call last):
>   File "/home/tito/sw_install/pycbc//bin/pycbc_live", line 4, in <module>
>     __import__('pkg_resources').run_script('PyCBC===073fb5', 'pycbc_live')
>   File "/home/tito/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 739, in run_script
>     self.require(requires)[0].run_script(script_name, ns)
>   File "/home/tito/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 1500, in run_script
>     exec(code, namespace, namespace)
>   File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/EGG-INFO/scripts/pycbc_live", line 512, in <module>
>     followup_ifos=usable_followup_ifos)
>   File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/EGG-INFO/scripts/pycbc_live", line 114, in check_coincs
>     bank, followup_ifos)
>   File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/EGG-INFO/scripts/pycbc_live", line 103, in compute_followup_data
>     check_state=(ifo in followup_ifos))
>   File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/pycbc/filter/matchedfilter.py", line 1783, in compute_followup_snr_series
>     return snr[onsource_slice] * norm, stilde.psd
>   File "<decorator-gen-120>", line 2, in __getitem__
>   File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/pycbc/types/array.py", line 65, in _convert
>     return fn(self, *args)
>   File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/pycbc/types/array.py", line 738, in __getitem__
>     return self._getslice(index)
>   File "/home/tito/sw_install/pycbc/lib64/python2.7/site-packages/PyCBC-073fb5-py2.7-linux-x86_64.egg/pycbc/types/timeseries.py", line 105, in _getslice
>     raise ValueError('Negative start index not supported')
> ValueError: Negative start index not supported
>
> ~tito/projects/pycbc/debug_2249 @ CIT.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/gwastro/pycbc/issues/2249#issuecomment-405618353>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/ACGrRip6hx9cm7Swy8ak0O19MLaUAYiYks5uHf7wgaJpZM4VRMqS>
> .
>



-- 
Dr. Alexander Nitz
Max Planck Institute for Gravitational Physics (Albert Einstein Institute)
Callinstrasse 38
D-30167 Hannover,  Germany
Tel: +49 511 762-17097

The weird thing is that I *am* allowing it to look back. However it appears that this trigger time is slightly *before* `data_reader.start_time`, so the lookback window is too short by ~20 samples or so:
```
2018-07-17 09:49:51,092 node578 Generating SEOBNRv4_ROM, 16.0s, 1590, starting from 26.6 Hz
2018-07-17 09:49:51,119 node578 [H1] trig_time - snr.start_time = 0.047852
2018-07-17 09:49:51,119 node578 [H1] trig_time - data_reader.start_time = 0.000488
2018-07-17 09:49:51,120 node578 onsource_slice = slice(1, 196, None)
2018-07-17 09:49:51,120 node578 window duration = 0.095000
2018-07-17 09:49:51,146 node578 [L1] trig_time - snr.start_time = 0.038574
2018-07-17 09:49:51,146 node578 [L1] trig_time - data_reader.start_time = -0.008789
2018-07-17 09:49:51,146 node578 onsource_slice = slice(-18, 177, None)
2018-07-17 09:49:51,146 node578 window duration = 0.095000
```
I am not sure how that can happen. I am looking at `LiveBatchMatchedFilter._process_batch()` but so far everything looks correct and it does not look like a simple rounding error either. I am sprinkling `_process_batch` with `prints()` and about to rerun, but let me know if you have any hint.

Resolving patch:
diff --git a/pycbc/filter/matchedfilter.py b/pycbc/filter/matchedfilter.py
--- a/pycbc/filter/matchedfilter.py
+++ b/pycbc/filter/matchedfilter.py
@@ -1696,7 +1696,8 @@ def _process_batch(self):
         return result, veto_info
 
 def compute_followup_snr_series(data_reader, htilde, trig_time,
-                                duration=0.095, check_state=True):
+                                duration=0.095, check_state=True,
+                                coinc_window=0.05):
     """Given a StrainBuffer, a template frequency series and a trigger time,
     compute a portion of the SNR time series centered on the trigger for its
     rapid sky localization and followup.
@@ -1725,15 +1726,15 @@ def compute_followup_snr_series(data_reader, htilde, trig_time,
         If True, and the detector was offline or flagged for bad data quality
         at any point during the inspiral, then return (None, None) instead.
 
+    coinc_window : float (optional)
+        Maximum possible time between coincident triggers at different
+        detectors. This is needed to properly determine data padding.
+
     Returns
     -------
     snr : TimeSeries
         The portion of SNR around the trigger. None if the detector is offline
         or has bad data quality, and check_state is True.
-
-    psd : FrequencySeries
-        The noise PSD corresponding to the trigger time. None if the detector
-        is offline or has bad data quality, and check_state is True.
     """
     if check_state:
         # was the detector observing for the full amount of involved data?
@@ -1743,7 +1744,7 @@ def compute_followup_snr_series(data_reader, htilde, trig_time,
         if data_reader.state is not None \
                 and not data_reader.state.is_extent_valid(
                         state_start_time, state_duration):
-            return None, None
+            return None
 
         # was the data quality ok for the full amount of involved data?
         dq_start_time = state_start_time - data_reader.dq_padding
@@ -1751,39 +1752,48 @@ def compute_followup_snr_series(data_reader, htilde, trig_time,
         if data_reader.dq is not None \
                 and not data_reader.dq.is_extent_valid(
                         dq_start_time, dq_duration):
-            return None, None
+            return None
 
     stilde = data_reader.overwhitened_data(htilde.delta_f)
 
     norm = 4.0 * htilde.delta_f * htilde.sigmasq(stilde.psd) ** (-0.5)
 
+    sr = data_reader.sample_rate
+    dt = 1. / sr
+
     qtilde = zeros((len(htilde) - 1) * 2, dtype=htilde.dtype)
     correlate(htilde, stilde, qtilde)
-    snr = qtilde * 0
+    snr = zeros((len(htilde) - 1) * 2, dtype=htilde.dtype)
     ifft(qtilde, snr)
 
-    valid_end = int(len(qtilde) - data_reader.trim_padding)
-    valid_start = int(valid_end - data_reader.blocksize * data_reader.sample_rate)
+    valid_end = int(len(snr) - data_reader.trim_padding)
+    valid_start = int(valid_end - data_reader.blocksize * sr)
 
-    half_dur_samples = int(data_reader.sample_rate * duration / 2)
-    valid_start -= half_dur_samples
+    half_dur_samples = int(sr * duration / 2)
+    coinc_samples = int(sr * coinc_window)
+    valid_start -= half_dur_samples + coinc_samples
     valid_end += half_dur_samples
     if valid_start < 0 or valid_end > len(snr)-1:
-        raise ValueError('Requested SNR duration ({0} s) too long'.format(duration))
+        raise ValueError(('Requested SNR duration ({0} s)'
+                          ' too long').format(duration))
 
-    snr = snr[slice(valid_start, valid_end)]
-    snr_dt = 1. / data_reader.sample_rate
-    snr_epoch = data_reader.start_time - half_dur_samples * snr_dt
-    snr = TimeSeries(snr, delta_t=snr_dt, epoch=snr_epoch)
+    epoch = data_reader.start_time - (half_dur_samples + coinc_samples) * dt
+    snr = TimeSeries(snr[slice(valid_start, valid_end)],
+                     delta_t=dt, epoch=epoch)
 
-    onsource_idx = int(round(float(trig_time - snr.start_time) * snr.sample_rate))
+    onsource_idx = int(round(float(trig_time - snr.start_time) * sr))
     onsource_slice = slice(onsource_idx - half_dur_samples,
                            onsource_idx + half_dur_samples + 1)
-    return snr[onsource_slice] * norm, stilde.psd
+    return snr[onsource_slice] * norm
 
 __all__ = ['match', 'matched_filter', 'sigmasq', 'sigma', 'get_cutoff_indices',
-           'sigmasq_series', 'make_frequency_series', 'overlap', 'overlap_cplx',
-           'matched_filter_core', 'correlate', 'MatchedFilterControl', 'LiveBatchMatchedFilter',
-           'MatchedFilterSkyMaxControl','MatchedFilterSkyMaxControlNoPhase','compute_max_snr_over_sky_loc_stat_no_phase', 'compute_max_snr_over_sky_loc_stat',
-           'compute_followup_snr_series','compute_u_val_for_sky_loc_stat_no_phase','compute_u_val_for_sky_loc_stat']
+           'sigmasq_series', 'make_frequency_series', 'overlap',
+           'overlap_cplx', 'matched_filter_core', 'correlate',
+           'MatchedFilterControl', 'LiveBatchMatchedFilter',
+           'MatchedFilterSkyMaxControl', 'MatchedFilterSkyMaxControlNoPhase',
+           'compute_max_snr_over_sky_loc_stat_no_phase',
+           'compute_max_snr_over_sky_loc_stat',
+           'compute_followup_snr_series',
+           'compute_u_val_for_sky_loc_stat_no_phase',
+           'compute_u_val_for_sky_loc_stat']
 
diff --git a/pycbc/io/live.py b/pycbc/io/live.py
--- a/pycbc/io/live.py
+++ b/pycbc/io/live.py
@@ -60,14 +60,14 @@ def make_psd_xmldoc(psddict, xmldoc=None):
     """Add a set of PSDs to a LIGOLW XML document. If the document is not
     given, a new one is created first.
     """
-    if xmldoc is None:
-        xmldoc = ligolw.Document()
-        root_name = u"psd"
-        Attributes = ligolw.sax.xmlreader.AttributesImpl
-        lw = xmldoc.appendChild(
-            ligolw.LIGO_LW(Attributes({u"Name": root_name})))
-    else:
-        lw = xmldoc.childNodes[0]
+    xmldoc = ligolw.Document() if xmldoc is None else xmldoc.childNodes[0]
+
+    # the PSDs must be children of a LIGO_LW with name "psd"
+    root_name = u"psd"
+    Attributes = ligolw.sax.xmlreader.AttributesImpl
+    lw = xmldoc.appendChild(
+        ligolw.LIGO_LW(Attributes({u"Name": root_name})))
+
     for instrument, psd in psddict.items():
         xmlseries = _build_series(psd, (u"Frequency,Real", u"Frequency"),
                                   None, 'deltaF', 's^-1')
@@ -95,8 +95,11 @@ def __init__(self, ifos, coinc_results, **kwargs):
         low_frequency_cutoff: float
             Minimum valid frequency for the PSD estimates.
         followup_data: dict of dicts, optional
-            Dictionary providing SNR time series and PSDs for each detector,
-            to be used in sky localization with BAYESTAR.
+            Dictionary providing SNR time series for each detector,
+            to be used in sky localization with BAYESTAR. The format should
+            be `followup_data['H1']['snr_series']`. More detectors can be
+            present than given in `ifos`. If so, the extra detectors will only
+            be used for sky localization.
         """
         self.template_id = coinc_results['foreground/%s/template_id' % ifos[0]]
 
@@ -107,7 +110,6 @@ def __init__(self, ifos, coinc_results, **kwargs):
         if 'followup_data' in kwargs:
             fud = kwargs['followup_data']
             self.snr_series = {ifo: fud[ifo]['snr_series'] for ifo in fud}
-            self.snr_series_psd = {ifo: fud[ifo]['psd'] for ifo in fud}
             usable_ifos = fud.keys()
             followup_ifos = list(set(usable_ifos) - set(ifos))
         else:
@@ -225,9 +227,10 @@ def __init__(self, ifos, coinc_results, **kwargs):
         outdoc.childNodes[0].appendChild(coinc_inspiral_table)
 
         # append the PSDs
+        self.psds = kwargs['psds']
         psds_lal = {}
-        for ifo in kwargs['psds']:
-            psd = kwargs['psds'][ifo]
+        for ifo in self.psds:
+            psd = self.psds[ifo]
             kmin = int(kwargs['low_frequency_cutoff'] / psd.delta_f)
             fseries = lal.CreateREAL8FrequencySeries(
                 "psd", psd.epoch, kwargs['low_frequency_cutoff'], psd.delta_f,
@@ -279,8 +282,8 @@ def upload(self, fname, gracedb_server=None, testing=True,
             for ifo in self.snr_series:
                 self.snr_series[ifo].save(snr_series_fname,
                                           group='%s/snr' % ifo)
-                self.snr_series_psd[ifo].save(snr_series_fname,
-                                              group='%s/psd' % ifo)
+                self.psds[ifo].save(snr_series_fname,
+                                    group='%s/psd' % ifo)
 
         gid = None
         try:
diff --git a/pycbc/types/timeseries.py b/pycbc/types/timeseries.py
--- a/pycbc/types/timeseries.py
+++ b/pycbc/types/timeseries.py
@@ -102,7 +102,8 @@ def _getslice(self, index):
             new_epoch = self._epoch
         else:
             if index.start < 0:
-                raise ValueError('Negative start index not supported')
+                raise ValueError(('Negative start index ({})'
+                                  ' not supported').format(index.start))
             new_epoch = self._epoch + index.start * self._delta_t
 
         if index.step is not None:


Test patch:
diff --git a/test/test_io_live.py b/test/test_io_live.py
--- a/test/test_io_live.py
+++ b/test/test_io_live.py
@@ -28,6 +28,7 @@
 from glue.ligolw import lsctables
 from glue.ligolw import table
 from glue.ligolw import utils as ligolw_utils
+from lal import series as lalseries
 
 # if we have the GraceDb module then we can do deeper tests,
 # otherwise just fall back to quicker ones
@@ -125,6 +126,13 @@ def do_test(self, n_ifos, n_ifos_followup):
                 read_coinc, lsctables.CoincInspiralTable.tableName)
         self.assertEqual(len(coinc_table), 1)
 
+        # make sure lalseries can read the PSDs
+        psd_doc = ligolw_utils.load_filename(
+                coinc_file_name, verbose=False,
+                contenthandler=lalseries.PSDContentHandler)
+        psd_dict = lalseries.read_psd_xmldoc(psd_doc)
+        self.assertEqual(set(psd_dict.keys()), set(all_ifos))
+
         shutil.rmtree(tempdir)
 
     def test_2_ifos_no_followup(self):