stbt.py: Fix debug message introduced in 5a0770e6

This was printing "/GstPipeline:pipeline0 (gst.Pipeline)" instead of the
textual pipeline.
stbt.py: Refactor: Move restartable source_pipeline code

So that the pipeline is parsed *after* the main application pipeline is
playing. This will make the next commit easier to understand. The
current commit only moves the code around; it doesn't make any other
changes except renaming the local variable "source" to
"restartable_source" for clarity.
stbt.py: "restart_source" behaviour chooses a free port

Instead of using GStreamer's udpsink/udpsrc default port of 4951.
This allows using multiple HD PVRs on the same PC at once.

We let the application pipeline's udpsrc pick its own port; we can
find out which port it picked by querying the "port" property once the
pipeline has reached at least the PAUSED state. See
http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gst-plugins-good-plugins/html/gst-plugins-good-plugins-udpsrc.html#gst-plugins-good-plugins-udpsrc.description

See commit 5a0770e6 for more details on "restart_source".
stbt.py: Trigger source restarts from underruns in the *application* pipeline

Not from underruns in the *source* pipeline. This should catch any
problems in the communication between the source and application
pipelines, not just problems within the source pipeline itself.

I don't think I've seen any actual problem that this solves, but it
seems more robust this way, and the code reflects more closely
the *goal* of the source restarting behaviour.
stbt.Display.frames: Ignore buffers with timestamp == MAX_UNSIGNED_INT

Occasionally, immediately after a restart† the HD PVR delivers *one*
frame with a correct timestamp (which sets "self.start_timestamp" here
in "frames") followed by one frame with a timestamp of
18446744073709551615 (2^64 - 1). This caused "frames" to time out
immediately.

"sys.maxsize" is the maximum *signed* integer, i.e. 2^(n-1) - 1.
The maximum unsigned integer is 2^n - 1 == 2 * (2^(n-1) - 1) + 1.

† See commit 5a0770e6.
tests: Increase timeouts

These tests are all expected to complete quickly, so the increased
timeout doesn't increase the run time of the test suite under normal
conditions. It does avoid some false failures when the system is under
heavy load.
Revert "Source-restart-on-video-loss behaviour is configurable, uses separate pipeline"

This reverts commit 5a0770e6, along with the fixes to 5a0770e6 in
d633efa8, 43de8794, 04728a52, 6f314688, and 54ac51a8.

Because 5a0770e6 isn't 100% reliable:

* The video stream sometimes displays a static image after the pipeline
  is restarted. I *think* this only affects the xvimagesink, because the
  application pipeline continues to process frames and finds, e.g. the
  UI main menu when I press the "MENU" button, even though I don't see
  it on the xvimagesink output. The application pipeline certainly does
  receive frames of some sort, judging from the debug output from
  "wait_for_match".

* Seeking in the video recorded by "stbt run --save-video" doesn't
  always work in the part of the video after a source restart.

* On a heavily loaded system the UDP transport (which is for getting
  frames from the source pipeline to the application pipeline) starts
  dropping keyframes so the received video is unusable for image
  comparison purposes.

* 5a0770e6 breaks the end-of-stream teardown introduced in b7a37493
  when the source device is a Blackmagic video-capture card (with
  "restart_source = False"). The output from "stbt run" is:

      stbt-run: teardown: Sending eos
      stbt-run: glib thread: reporting exception to user thread:
          Internal GStreamer error: negotiation problem.
      gstdecklinksrc.cpp(1248): gst_decklink_src_task ():
          /GstPipeline:pipeline0/GstDecklinkSrc:decklinksrc0

  whereupon "stbt run" exits immediately and the video index is not
  written so the saved video is not streamable.

Maybe the solution to the first 3 points will be to send raw video over
a protocol like RTP, and the solution to the last point will be to put
the decklinksrc in a separate bin, like the pre-5a0770e6 implementation
even though the separate bin isn't used for the source restarting
mechanism. But I don't have time to pursue this any further, so for now
I'm reverting this.
stbt.py: Source-restart-on-video-loss behaviour is configurable

Commit 5a0770e6 made this behaviour configurable *and* also changed the
way source restarting is done. 5a0770e6 was reverted in a189e0c9 because
the new way of restarting the source was unreliable. This current commit
applies the configurability changes from 5a0770e6 without changing the
source restarting mechanism.

We want this to be configurable because some video sources don't *need*
this behaviour. Only the Hauppauge HD PVR needs it. With the Blackmagic,
for example, we prefer to block until video is re-established (or until
"stbt.Display.get_frame" times out) because it is simpler, the logs are
less polluted with "stopping"/"starting" messages, and if you're
recording a video with "--save-video" the video only contains what
happened since the last restart.

Note that the config value accepts "1", "yes", "true", or "on" (case
insensitively), unlike 5a0770e6 which only accepted "True". This is
consistent with ConfigParser.getboolean -- unfortunately our
"get_config" wrapper around ConfigParser doesn't allow us to call
getboolean directly.
stbt-completion: Fix completion of "--save-video"

This was missing from 2ee7b00.
stbt.py: Cleanup: Use "1e9" instead of "1000000000"

It's easier to read, and less likely to have an incorrect number of
zeros. "1e9" is a floating-point value instead of an integer, but here
it doesn't matter.
stbt.Display.frames: Debug message showing timestamp when timed out

In case we see the behaviour described in d633efa8 again after a189e0c9.
stbt.Display.restart_source_bin: Don't sleep in the GLib main loop's thread

So that pressing Control-C while the pipeline is being restarted will
(most likely) be caught by the main application thread, not the GLib
thread, and thus the application will actually exit.
stbt.Display.teardown: Avoid intermittent error from decklinksrc

Occasionally, on teardown the following GStreamer error would happen
*before* the pipeline received the EOS event, so if a video was being
recorded it wouldn't have the index information necessary for seeking.

    Internal GStreamer error: negotiation problem.
    gstdecklinksrc.cpp(1248): gst_decklink_src_task ():
      /GstPipeline:stb-tester/GstBin:bin0/GstDecklinkSrc:decklinksrc0

It is still necessary to post an EOS message to the source bin, even
after unlinking it from the sink bin (see b7a37493).

I tested this with:

    while ./stbt-run --save-video video.webm -vv test.py &&
          webminspector.py video.webm | grep 'Cue Point';
    do
        :
    done

which would fail after 3 or 4 runs before this commit, and succeeded 50
times with this commit. See b7a37493 for details on webminspector.py.
stbt.Display.teardown doesn't hang when No Video

When NoVideo was raised at the very beginning of the test, "teardown"
completed without error but the EOS event was never received from the
pipeline, so "on_eos" was never called, so the application didn't exit
because the GLib main loop's thread was a non-daemon thread.[1]

[1] http://docs.python.org/2/library/threading.html#thread-objects

When NoVideo was raised halfway through a test (video was initially
available but was then lost), "teardown" raised the following error
(presumably because "self.source_bin" was "None" because we were halfway
through a source restart) and the application also never exited:

    stbt-run: teardown: Sending eos
    Traceback (most recent call last):
      File "./stbt-run", line 50, in <module>
        stbt.teardown_run()
      File "/home/stb-tester/stb-tester/stbt.py", line 648, in teardown_run
        _display.teardown()
      File "/home/stb-tester/stb-tester/stbt.py", line 858, in teardown
        gst.element_unlink_many(self.source_bin, self.sink_bin)
    TypeError: argument must be a GstElement

This commit fixes both cases, by remembering if NoVideo was raised. We
mark the GLib main loop's thread as a daemon thread, and (if there *is*
video) we explicitly wait for the thread to complete (with a timeout
just in case anything unforeseen goes wrong).
rpm spec file: "BuildArch: noarch"

To reflect the reality since ca4201df.
tests/test-stbt-py.sh: Add "--verbose" to all "stbt-run" invocations

This makes it easier to debug failing tests, and is how "stbt-run" is
invoked in all our other tests.
stbt.py: Refactor: Separate source & sink pipelines, joined with app{sink,src}

This will allow the caller of `stbt.Display.frames` to modify the
GStreamer buffer (the video frame) before passing it to the sink
pipeline. The next commit will do just that.

This is very similar to 5a0770e6's change to the source restarting
mechanism (which was reverted in a189e0c9) but using appsink/appsrc
instead of udpsink/udpsrc.

The implementation is as follows. We split the GStreamer pipeline into
two separate pipelines: A "source pipeline" ending in an appsink, and a
"sink pipeline" starting with an appsrc.

    -source pipeline--   on_new_buffer/frames   -sink pipeline--
    | ... ! appsink  |  --------buffer------->  | appsrc ! ... |
    ------------------                          ----------------

Every time a new buffer is available on the source pipeline's appsink,
we push it to the sink pipeline's appsrc (see `Display.on_new_buffer`).

But when we're inside `Display.frames` (i.e. inside `wait_for_match` or
similar) we *don't* want to display new frames, we only want to display
the frames processed by `wait_for_match`. Note that the `wait_for_match`
image-processing algorithm is slow, so it only processes one out of
every few frames (and we drop the rest). If we displayed the unprocessed
frames in between the frames that `wait_for_match` *did* process, the
user wouldn't see the `wait_for_match` visualisation (see the next
commit) because it would only be present in one out of every few frames.

So we use a lock to disable `on_new_buffer` while we're inside `frames`.
Note that `frames` calls arbitrary user code (via `yield`) which may
throw an exception; thus exceptions from `frames` are not necessarily
fatal and may even be caught in the calling user code. So it's important
that `frames` always releases the lock.

Note that `stbt.get_frame` can be safely called at any time -- we don't
need to lock there, because `Display.last_frame` has its own locking
built-in. `Display.on_new_buffer` pushes the buffer to
`Display.last_frame` even when the lock is held by `Display.frames`, in
case the user calls `stbt.get_frame`.

Note that `Display.init` keeps a reference to `sink_pipeline`, even
though it isn't referenced elsewhere, just to keep it alive.
stbt.py: Match & motion visualisation

This fixes the regression introduced in ca4201df.

Match: Show the red/yellow "wait_for_match" rectangle.

Motion: The visualisation has changed slightly from the pre-ca4201df
visualisation. Instead of flashing the whole screen red, just highlight
in red the areas where motion is detected. Note that we dilate the area
just to make it slightly more obvious. Also unlike the pre-ca4201df,
we don't darken the masked-out areas. Instead, the mask will be obvious
if there is any motion passing through the masked-out area.

Implementation notes
--------------------

I'm not 100% sure whether or not we're copying buffers more than is
necessary.

I *think* that we're *not* copying data from the GStreamer buffer into
the OpenCV/numpy array:

    >>> image = gst_to_opencv(gst_buffer)
    >>> print image.flags
    OWNDATA : False
    WRITEABLE : False
    ...

Compare with:

    >>> image2 = image.copy()
    >>> print image2.flags
    OWNDATA : True
    WRITEABLE : True
    ...

However:

    >>> print id(gst_buffer.data)
    4418195456
    >>> print id(image.base)
    4418195456
    >>> print id(image.data)
    4418004912

What is "image.data" and why is it different to "gst_buffer.data"?

It *is* possible to modify the OpenCV image (e.g. with "cv2.rectangle").
The changes aren't reflected in the GStreamer buffer.

    >>> cv2.rectangle(image, pt1, pt2, color)
    >>> cv2.imwrite("image.png", image)
    >>> appsrc.emit("push-buffer", gst_buffer)
    >>> print id(image.base)
    4418195456
    >>> print id(image.data)
    4418004912

Note that "image.data" is still pointing at the same location in memory,
so there doesn't seem to be any copy-on-write going on. Maybe we *did*
do a copy from the GStreamer buffer when we created the OpenCV/numpy
image? I know that the rectangle *is* present in the OpenCV image,
because it's there when I open "image.png" in an image viewer. The
rectangle isn't present in the xvimagesink that is showing the GStreamer
buffers from appsrc.

I also couldn't figure out how to modify the GStreamer buffer data
directly. I tried this:

    >>> data = memoryview(gst_buffer.data)
    >>> data[0] = 0
    TypeError: cannot modify read-only memory

"memoryview" is the python interface to directly reading or writing
memory without copying. It only works on data structures implemented in
a python C extension where the data structure implements the python C
api's "buffer" interface.

I couldn't even modify the memory of a new buffer, where the memory
is allocated by GStreamer:

    >>> newbuf = gst.buffer_new_and_alloc(gst_buffer.size)
    >>> data = memoryview(newbuf.data)
    >>> data[0] = 0
    TypeError: cannot modify read-only memory

I don't know if this is because the GStreamer python bindings don't
support python's memoryview/buffer interface, or because GStreamer has
marked this memory as read-only.

Note that memoryview will fail if given an object that doesn't implement
the buffer interface:

    >>> memoryview([1,2,3,4,5])
    TypeError: cannot make memory view because object does not have the
               buffer interface

However, it could be that GStreamer is just providing a python string,
which is immutable:

    >>> m = memoryview("12345")
    >>> m
    <memory at 0x11070f510>
    >>> m[0] = 0
    TypeError: cannot modify read-only memory

In the end I had to create a new GStreamer buffer pointing at the OpenCV
image:

    >>> newbuf = gst.Buffer(image.data)
    >>> newbuf.set_caps(gst_buffer.get_caps())
    >>> newbuf.timestamp = gst_buffer.timestamp
    >>> self.appsrc.emit("push-buffer", newbuf)

This works perfectly so far. I don't know if this copies the data, or
merely points at it. If it doesn't copy the data, I don't understand who
owns the image.data memory, but it seems to survive unharmed all the way
to the xvimagesink and to the recorded video, so maybe the GStreamer
python bindings take ownership of it in such a way that python doesn't
garbage-collect it.

I'm satisfied that this is good enough for now. Performance seems much
the same as before, but I haven't done any measurements; I'm just
estimating based on the visible frame rate. The `wait_for_match`
algorithm is so slow that it dwarfs any performance effects from this
commit. However I might try to understand the copying (or not) behaviour
more thoroughly when I port to GStreamer 1.0.
stbt.Display.frames: Display the last frame processed

When `frames` is called from `wait_for_match`, this is usually the frame
that contains the match.

(Once a match is found, `wait_for_match` stops iterating over the
`frames` generator, so the code after the `yield` wasn't being run for
the frame with the match. The `finally` block presumably gets run when
the generator object is destroyed -- in the CPython implementation this
will be when `wait_for_match` returns.)
stbt.Display.__init__: Move member variables to the top of the function

`Display.on_new_buffer` was failing with:

    AttributeError: Display instance has no attribute 'lock'

Signal handlers (such as `Display.on_new_buffer`) are called
synchronously from the thread of the GStreamer element in question (in
this case, from the `queue` thread). So these signal handlers can be
entered even before we start running the GLib main loop (which is only
needed for receiving GStreamer bus *messages*).