Repository: gwastro/pycbc

Issue:
PyCBC Live error in followup code when using state/DQ channels
```
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/pycbc.live/testing/O2_replay/pycbc_live_O2_replay_env/lib/python2.7/site-packages/mpi4py/__main__.py", line 7, in <module>
    main()
  File "/home/pycbc.live/testing/O2_replay/pycbc_live_O2_replay_env/lib/python2.7/site-packages/mpi4py/run.py", line 196, in main
    run_command_line(args)
  File "/home/pycbc.live/testing/O2_replay/pycbc_live_O2_replay_env/lib/python2.7/site-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "/usr/lib64/python2.7/runpy.py", line 240, in run_path
    return _run_module_code(code, init_globals, run_name, path_name)
  File "/usr/lib64/python2.7/runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/pycbc.live/testing/O2_replay/pycbc_live_O2_replay_env/bin/pycbc_live", line 607, in <module>
    data_reader, bank)
  File "/home/pycbc.live/testing/O2_replay/pycbc_live_O2_replay_env/bin/pycbc_live", line 174, in check_coincs
    bank, followup_ifos)
  File "/home/pycbc.live/testing/O2_replay/pycbc_live_O2_replay_env/bin/pycbc_live", line 134, in compute_followup_data
    data_readers[ifo], bank, template_id, coinc_times)
  File "/home/pycbc.live/testing/O2_replay/pycbc_live_O2_replay_env/lib/python2.7/site-packages/pycbc/filter/matchedfilter.py", line 1768, in followup
_event_significance
    if not data_reader.state.is_extent_valid(state_start, state_end):
  File "/home/pycbc.live/testing/O2_replay/pycbc_live_O2_replay_env/lib/python2.7/site-packages/pycbc/frame/frame.py", line 736, in is_extent_valid
    e = s + int(duration * sr) + 1
RuntimeError: Internal function call failed: Input domain error
```


Discussion:
I'm guessing input domain error is from reading data beyond the end of the frame?
@titodalcanton In this case, was the analysis running behind? And the frame disappeared?
@titodalcanton Wait scratch that. The line it points to in the traceback is. 
https://github.com/gwastro/pycbc/blob/master/pycbc/frame/frame.py#L736
There is no frame reading here. It's already happened. There's also nothing that could return an Input domain error.

Is it possible the traceback was not related to the error that followed in the log?  
Input domain error is something that either lalframe or lalsimulation is returning. I don't think anything else can produce this. I don't see where the associated call would be though...
Ah, I was wondering if it might be due to LIGOTimeGPS. Here is another case with a few more preceding lines.
```
2019-03-09 09:03:18,029 node742 Found V1-H1 coinc with ifar 0.000179487001744                                                                [29/1956]
2019-03-09 09:03:18,029 node742 computing followup data for coinc
2019-03-09 09:03:18,030 node742 Generating SPAtmplt, 96.0s, 1613, starting from 25.1 Hz
XLALGPSSetREAL8(): overflow 19778866910XLAL Error - XLALGPSSetREAL8 (XLALTime.c:83): Input domain error
XLAL Error - XLALGPSMultiply (XLALTime.c:281): Internal function call failed: Input domain error
Traceback (most recent call last):
  File "/usr/lib64/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/mpi4py/__main__.py", line 7, in <module>
    main()
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/mpi4py/run.py", line 196, in main
    run_command_line(args)
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "/usr/lib64/python2.7/runpy.py", line 240, in run_path
    return _run_module_code(code, init_globals, run_name, path_name)
  File "/usr/lib64/python2.7/runpy.py", line 82, in _run_module_code
    mod_name, mod_fname, mod_loader, pkg_name)
  File "/usr/lib64/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/tito/projects/pycbc/live/multidetector/env/bin/pycbc_live", line 607, in <module>
    data_reader, bank)
  File "/home/tito/projects/pycbc/live/multidetector/env/bin/pycbc_live", line 174, in check_coincs
    bank, followup_ifos)
  File "/home/tito/projects/pycbc/live/multidetector/env/bin/pycbc_live", line 134, in compute_followup_data
    data_readers[ifo], bank, template_id, coinc_times)
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/pycbc/filter/matchedfilter.py", line 1766, in followup_event_sign
ificance
    if not data_reader.state.is_extent_valid(state_start, state_end):
  File "/home/tito/projects/pycbc/live/multidetector/env/lib/python2.7/site-packages/pycbc/frame/frame.py", line 744, in is_extent_valid
    e = s + int(duration * sr) + 1
RuntimeError: Internal function call failed: Input domain error
```
The error is not so surprising given this:
```
In [8]: lal.GPSTimeNow() * 2
XLALGPSSetREAL8(): overflow 2472373988XLAL Error - XLALGPSSetREAL8 (XLALTime.c:83): Input domain error
XLAL Error - XLALGPSMultiply (XLALTime.c:281): Internal function call failed: Input domain error
---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
<ipython-input-8-7e2462e0ef3b> in <module>()
----> 1 lal.GPSTimeNow() * 2

RuntimeError: Internal function call failed: Input domain error
```
Ah, this looks like a wrong call to `is_extent_valid()`. The second argument should be a duration, not the end time, right?

https://github.com/gwastro/pycbc/blob/master/pycbc/filter/matchedfilter.py#L1766

Resolving patch:
diff --git a/pycbc/filter/matchedfilter.py b/pycbc/filter/matchedfilter.py
--- a/pycbc/filter/matchedfilter.py
+++ b/pycbc/filter/matchedfilter.py
@@ -1761,16 +1761,17 @@ def followup_event_significance(ifo, data_reader, bank,
 
     # Require all strain be valid within lookback time
     if data_reader.state is not None:
-        state_end = data_reader.strain.end_time - data_reader.reduced_pad
-        state_start = state_end - lookback
-        if not data_reader.state.is_extent_valid(state_start, state_end):
+        state_start_time = data_reader.strain.end_time \
+                - data_reader.reduced_pad - lookback
+        if not data_reader.state.is_extent_valid(state_start_time, lookback):
             return None, None, None
 
     # We won't require that all DQ checks be valid for now, except at
     # onsource time.
     if data_reader.dq is not None:
-        if not data_reader.dq.is_extent_valid(onsource_start - duration / 2.0,
-                                              onsource_end + duration / 2.0):
+        dq_start_time = onsource_start - duration / 2.0
+        dq_duration = onsource_end - onsource_start + duration
+        if not data_reader.dq.is_extent_valid(dq_start_time, dq_duration):
             return None, None, None
 
     # Calculate SNR time series for this duration


Test patch: