Ignore templatematch messages received before appsink has received any frames

We have an appsink (self.screenshot in Display.bus_message) that we use
to determine the current timestamp of the stream. We calculate the
wait_for_match timeout from that timestamp.

The very first templatematch messages can arrive before the appsink has
received any frames, which caused the following error:

      File "/usr/local/lib/stbt/stbt.py", line 229, in bus_message
        self.start_timestamp = buf.timestamp
    AttributeError: 'NoneType' object has no attribute 'timestamp'

This error happens during a callback (to Display.bus_message), and
presumably the exception is caught by the callback dispatcher mechanism
(the gtk.main loop), because it didn't stop `stbt run` from proceeding.
This commit merely removes the distracting error message from the logs.
Cleanup: Move unit-test infrastructure to end of file
Cleanup: Rename 'bus_message' callback function to 'on_match'

It is clearer for this function's name to reflect what it does, rather
than when it is called. Particularly important now that we have several
message handlers.
Wrap the source pipeline in a "source bin"

This is in support of future changes to recover from loss of input video
signal.

A gstreamer "bin" is a pipeline (of several gstreamer elements) that can
be treated like a single element.

Note that gst.parse_bin_from_description(_, True) tells it to create the
"ghost pads" -- the pads of the bin as a whole, connected to a pad of a
particular element in the bin. This works when creating the sink bin
because its elements only contain a single unconnected sink pad (the
sink of the left-most queue).

The pipeline of the source bin is outside our control (because it is
configurable) and it might have multiple source pads, so we create the
ghost pad manually.

The source pipeline might not know all of its pads yet: For example
decodebin (used in our source pipeline for the Hauppauge HDPVR capture
device) doesn't know its source pad until it knows the format of the
stream it will be decoding, i.e. until the pipeline has started playing.
When we were using gst.parse_launch, it (presumably) was registering a
handler for decodebin's "new-decoded-pad" signal, and linking the pads
in that handler. Instead of doing that ourselves, we add a capsfilter
element to force decodebin to create a source pad with the specified
caps.
Restart source_bin whenever input video signal is lost

Without this, the pipeline never recovers from a loss of input signal,
even after the input signal to the capture device is re-established.

The symptoms, in chronological order, were:

* The Hauppauge HDPVR capture device lost its input signal
* The video on screen turned completely blue
* The pipeline continued to receive and process a few frames (15 or so)
  of blue screen.
* The Hauppauge's light turned off and no further frames received by
  the pipeline. The X Window with the blue screen stays up on screen.
* After the video input to the Hauppauge is re-established, the
  Hauppauge's light never switched back on. The X Window with the blue
  screen is still up on screen and the pipeline is stuck. `stbt run`
  never terminates.

A momentary loss of input signal can be caused by:

* A stbt script that restarts the set-top box.
* A stbt script that causes an HDMI/HDCP renegotiation (e.g. by changing
  between SD and HD channels on a set-top box with a buggy HDMI/HDCP
  implementation).

SOME NOTES ON THE IMPLEMENTATION:

`restart_source_bin` is run in the main application thread (the glib
thread) because it is scheduled from a glib timer; so it can safely shut
down the gstreamer pipeline elements. On the other hand, `on_underrun`
is run from the queue element's streaming thread; and you cannot change
the state of an element from its streaming thread.

`threads_init` is to avoid a segfault in `on_underrun` with the error
message "Fatal Python error: GC object already tracked". This solution
is recommended by a StackOverflow answer.[1]

After receiving an underrun signal from the queue, we wait for one
second before calling `restart_source_bin`, and if the queue receives
further buffers in that time, we cancel `restart_source_bin`. This is
because we always get one `underrun` signal when starting the pipeline
for the first time and after restarting the source bin.

The signal from the queue telling us that it is receiving further
buffers is "running"[2]. We chose "running" (has received more than some
min-threshold of frames) rather than "pushing" (has received a single
frame) to add some extra hysteresis to the system. We might want to
revisit this decision now that we are only receiving a single underrun
after restarting the source bin (during development of this commit, a
different implementation of `restart_source_bin` was causing a severely
reduced frame rate after restarting, and the queue was continuously
emitting underrun signals).

Note that glib and gtk modules are both provided by the pygobject
package. At some point, several functions were moved from gtk to glib,
because they are not UI-specific. For example, gtk also has add_timeout
but it is deprecated; we use glib.add_timeout instead. gtk.threads_init
is deprecated in favour of gtk.gdk.threads_init; but the latter also
initialises gdk threads that we don't use, so we really should use
glib.threads_init instead, but glib.threads_init isn't available on
ScientificLinux6's version of pygobject2 (2.20.0).

SOME OTHER THINGS WE TRIED:

Setting sink_bin to PAUSED instead of READY[3] caused a really slow
framerate (2-3 fps) after re-gaining video signal.

Just setting `pipeline` to PLAYING without first setting `source_bin`
PLAYING caused the video to freeze as soon as the source_bin was
restarted. The pipeline continued to receive frames (as evidenced by the
debug output from `on_match`), but the image was frozen at the first
frame received after restarting the pipeline.

Just setting the source bin to null without completely removing and
re-creating it caused a severe latency problem: The pipeline would
restart, but the image on screen would remain blue until the pipeline
had run (after re-starting) for the same amount of time that it had been
running before the video loss. Presumably the source bin was somehow
reporting that the new video was part of the same original stream, and
the sink bin was waiting for stream timing information that came after
the frames it had already seen.

Just setting the entire pipeline to null and then playing: The X window
disappears, the pipeline seems to start running but the X window never
reappears.

[1] http://stackoverflow.com/questions/7496629/gstreamer-appsrc-causes-random-crashes/7498503#7498503

[2] http://gstreamer.freedesktop.org/data/doc/gstreamer/stable/gstreamer-plugins/html/gstreamer-plugins-queue.html#GstQueue-running

[3] http://gstreamer.freedesktop.org/data/doc/gstreamer/stable/manual/html/section-elements-states.html
Continue re-starting the source bin if it wasn't available

For example if the video signal didn't return before the 5 second sleep.

Note that the Hauppauge takes a few seconds to initialise itself once it
detects an input signal; starting the pipeline before the Hauppauge is
ready gives the error "libv4l2: error getting pixformat: Bad address".

We need to do this because the above libv4l2 error isn't reported to the
application (see the call to v4l2_fd_open in gst_v4l2_open in
gst-plugins-good/sys/v4l2/v4l2_calls.c).

This doesn't seem to work when the video signal has been unavailable for
too long (more than 1 or 2 attempts at `restart_source_bin`): After
video signal is re-established, when the source bin in restarted the
Hauppauge light turns on but the window stays blue and no frames are
received by the pipeline.
Fix infinite loop when a failing test ends

When a test fails, the call to `bus.poll` in `save_frame` never returns
(in spite of the timeout parameter).

The culprit was the call to gtk.gdk.threads_init() which we added in
3811764 to avoid a segfault in the queue's `underrun` signal handler.
This initialises gdk threads as well as any glib threads. Calling
glib.threads_init() instead, fixes this problem. I don't really
understand why and I hope it isn't just a race condition whose timing
has been shifted slightly.

Note that version of the glib python module on Scientific Linux 6
(pygobject 2.20.0) doesn't have `threads_init`, but the gobject module
does. These two modules are essentially the same thing, and using
gobject.{thread_init,add_timeout,source_remove} (instead of the same
functions in glib) works on SL6 and Fedora 17 (pygobject 2.28.6).
stbt-record `--control-recorder` help string

It seems to have been copy-and-pasted from the help string for
`--control`. This wording taken from the description in README.rst.
stbt-record cleanup: Use explicit "stbt" module qualifier
Cleanup: Add "t." to the beginning of *every* tee branch

There's clarity in symmetry.