packaging: Move all "common" into private package _stbt

With the dockerized tools coming up in stb-tester we want to use some of
the stb-tester functionality without incurring a dependency on the rest.
e.g. stbt service needs get/set_config but doesn't need a dependency on
GStreamer or OpenCV.  This has not been possible as all our code was in
a single `stbt.py` file.  The re-organisation 8ae1c0c split `stbt.py` up
into multiple files, but you still couldn't `import stbt.config` without
incurring a dependency on GStreamer.

This commit splits the stbt package into a private package `_stbt` with
no hard dependencies and a public one `stbt` with all of them.  Our
internal tools can now `import _stbt.config` etc. without hacks to
`PYTHONPATH`, etc to get import resolution to work.  It has the secondary
beneficial effect that our users won't think that `import stbt.config` is
a valid thing to do.  We still only officially support `import stbt`.
packaging: Rename `config._set_config` to `config.set_config`

`set_config` is internal to stbt, not internal to config.py.
Set sink_pipeline=xvimagesink in system config rather than as a default

To help support testing on headless systems.
Add Dockerfile

[Docker] is a implementation of Linux containers which offer many of the
same benefits as virtual machines, but are much more lightweight.  They
allow you to package up a program and it's dependencies such that it will
run exactly the same on any machine.  They can also provide isolatoion such
that the host machine is isolated from changes made in the container.

This is potentially very useful to stb-tester.  stb-tester can be difficult
to deploy due to the dependencies.  Docker offers a way to easily provide a
standardised stb-tester environment.  It is also useful in a CI context as,
much like Travis, you could run arbritary user code without risking the
host system.

A Dockerfile contains the instructions on how to create a container.
Containers start out with a base image and build on top of that.  In this
case the stb-tester container is based ubuntu 14.04 (LTS).  Any container
can be used as the base for other containers.  Our users can base their
containers on the stb-tester container if they wish.

Docker also provides an [index].  This is a global repository of docker
images.  Docker containers in the index are easier to deploy[^1] and use as
a base for other images.  It also provides a service called "[trusted
builds]" where your image will be built and published for you, from github,
with a guarantee that the sources there are the source of the container.
One constraint is that the Dockerfile must be in the root of the repo and
present at clone time.  This means that we can't generate it as a part of
the build process.

[^1]: You can just type `docker run stb-tester:latest` and you're up and
      running.
[Docker]: http://docker.io/
[index]: https://index.docker.io/
[trusted builds]: http://blog.docker.com/2013/11/introducing-trusted-builds/
Makefile: Add docker-build target

Creates a docker image from the current status in git.  This is useful if
you want a docker image of a development version of stb-tester, rather than
using one of the "[trusted builds]".

When building a Dockerfile docker keeps a cache of intermediate stages.  It
decides if it needs to re-run a command from the Dockerfile if the command
changes, or in the case of the ADD command, if the files that are being
added to the container change.  These changes include file metadata such as
mtime and atime.  I use `touch` here to normalise the timestamps of all the
files from the tarball to the UNIX epoch.  This avoids defeating docker's
caching so we don't have to download all the package dependencies every
time we type `make docker-build` as long as the list of dependencies don't
change.

[trusted builds]: http://blog.docker.com/2013/11/introducing-trusted-builds/
virtual-stb: Add virtual stb - useful for testing virtualized STBs

For instance to test an STB UI written for HbbTV or HTML5 you can use this
with chrome.
WIP: virtual-stb: Add readiness notification

TODO:
 - Add create notify socket, add fork() and print results
make docker-build: Also tag image with "stb-tester:makefile"

For convenience when testing changes to this stb-tester docker image in
another docker image whose Dockerfile can say "FROM stb-tester:makefile"
-- before this we had to edit that other Dockerfile after each change
to stb-tester.
virtual-stb/service/Dockerfile: This is a combination of 85 commits.

fixup! Add Dockerfile

fixup! Makefile: Add docker-build target

extra/virtual-stb: Add unit test for netstat output parsing

This test doesn't run the real netstat so it doesn't prove that the
function works in the real world, but at least it documents the parsing
code and what the expected input looks like.

Dockerfile: git is now a runtime dependency of stb-tester

Dockerfile: Set stb-tester user's $HOME

I don't know why it isn't set properly inside the container.

fixup! extra/virtual-stb: Work around netstat limitations in Docker container

WIP: Fix virtual-stb shutdown

TODO:
* Better commit message

extra/virtual-stb: Rename README => README.md

So that the github web interface displays it nicely.

fixup! virtual-stb: Add readyness notification

virtual-stb: Pull stbt config out into seperate Python file

Add make install-virtual-stb

Very few dependencies

fixup virtual-stb: Ensure .X11-unix exists and has right permissions

Fixes virtual-stb on systems not already running X

WIP: Add stbt-ssh-endpoint

Intended to allow creating a stateless machine which can be used by other
machines to run tests to help integrate stb-tester into other CI systems.

TODO:
* Document
* Test
* Combine with main Makefile
* Package-up

virtual-stb & service: Share timezone with Docker containers

Otherwise the time is incorrect.  I'm not certain whether this is a good
idea.  We should really be storing all timestamps as UTC and convert at
the edges via javascript or dynamically generated web-pages.  It still
leaves open the question of should we be sending a timezone to the test-
packs at all?

Also: perhaps a virtual-stb would want to have a different timezone.  This
needs to be solved before the first release of the service stuff as any
incompatible change there would be strictly forbidden.

fixup! WIP: Add stbt-ssh-endpoint

fixup! extra/service: Add web server container for displaying results

fixup! WIP: Add stbt-ssh-endpoint

fixup! fixup! WIP: Add stbt-ssh-endpoint

fixup! WIP: Add stbt-ssh-endpoint

REMOVEME: stbt-ssh-endpoint: Add command setup-lirc

This command should be removed once we figure out how to handle node
configuration properly and should definitely not make it into any
official releases.

fixup! REMOVEME: stbt-ssh-endpoint: Add command setup-lirc

extra/service/run: Fix pylint & nose errors

virtual-stb: Add run a fake-lircd in the virtual stb to inject key events

Allows customising the mapping between the keycode names that stbt sees and
the XKeySyms that X and the program under test understands.

Dockerfile: Load config file from tests repo

service: The system configuration should come from the host

service: Use the volumes from the virtual-stb

virtual-stb: Attempt to fix volumes

Fix add with-lircd to service

virtual-stb: Attempt to fix volumes

service: fixup

fix notification

vstb-lircd fixes

fix lirc socket path

vstb: Fix /proc/unix parsing code

Fixup lirc in virtual-stb

virtual-stb: Refactor so it can be used as a library by "service"

virtual-stb: Rename so it can be imported by other python files

service: fix using virtual_stb as a library

virtual-stb fixes

Add example virtual-stb

Build out example test

A better name for new example virtual-stb

Better name for test pack

Fix service/virtual-stb/Dockerfile and add test

service: Tidy-up

service: Fix Makefile/installation path

service: Fix make install-service

service: Fix paths

service: Fix direcotry

Fix service install direcotry

service: Fix PYTHONPATH

service: Another fix

service: Fix PYTHONPATH

virtual-stb: Fix and add test for custom key mappings

service tests: Move html5 test pack to make room for more

Add test for non-virtual-stb service run

service: Support running against system configured video capture

stbt virtual-stb: Fix handling of "--docker-image" arg

This fixes an error introduced in recent refactorings related to
ssh-endpoint / service.

stbt virtual-stb: Expose lirc via IP, not Unix-domain socket

This way it's easier to run "stbt run" on the host against a virtual-stb
running in a docker container (as opposed to when "stbt run" is also
running inside a docker container, which worked because the 2 containers
shared the /run/lirc/lircd socket).

fixup service/virtual-stb

Fixup service: Allow different paths to tests-pristine between host and container

Dockerfile/service: Add test-pack-entrypoint

test-pack-entrypoint works around some issues with running test-packs in
docker.

service: fixup stbt-ssh-endpoint

virtual-stb: fix running without docker

extra/service/serve-results: Fix .log mimetype, enable directory listing

Dockerfile: Refactor `make install`

Dockerfile: Don't `ENV DEBIAN_FRONTEND noninteractive`

It's better to export it in the individual rules as otherwise it ends up
in the environment of the commands run.

service: Add command to regenerate authorized_keys file for ssh login

Add Dockerfile for service

Fixup service

service: The docker command is `docker` not `docker.io`

We require docker >1.0 which even on Debian/Ubuntu uses the command
`docker`.

Fixup stbt service running in docker

fix make docker-build

rename service-users-volume

Fix make docker-publish

fixup docker

service: Add documentation

service README: Fix image

service README: Fix whitespace

service README: HTMLise table

github can't do multi-line table cells

fixup service & virtual-stb stuff (validating service installation instructions)

service run: Avoid `git checkout` every run

fixup service

service run: Avoid unnecessary git checkout

Dockerfile.service: Speed up docker build

...by caching build dependencies. This comes at the cost of a larger
docker image.

service run: TODO: pull/reset latest changes if "revision" is a branch

fixup service

service: Fix concurrent running of tests

service: Add a bunch of tests

fixup service

fixup service

extra/virtual-stb: Don't hardcode python location

So that when I run `stbt virtual-stb` on my Mac, it uses the python from
macports instead of the system python (otherwise python won't be able to
import all my third-party python libraries that I installed with
macports).

virtual-stb: Show virtual-stb.log if it failed to start

virtual-stb: Document why we use `os._exit`

virtual-stb: Flush std{out,err} before exiting

A typical usage pattern is:

    export $(stbt virtual-stb --daemonize --docker-image xxx)

to export the environment variables `VSTB_DOCKER_CID` etc, which are
printed to stdout by `stbt virtual-stb`. However, since with
`--daemonize` we use `_exit` instead of `exit`, this would
intermittently fail to work because `stbt` exited before anything was
printed.

docker: Fix `make docker-publish` and make more flexible

This allows you to customise the prefix and tag of the docker build
allowing publishing docker images to the docker hub under your username.
e.g.:

    make docker-publish docker_prefix=wmanley/ docker_tag=next

docker: Use stbtester as the docker prefix rather than stb_tester

I have registered the stbtester organisation on the docker registry.
Underscores are not allowed in the names.
WIP: Add "next" and "prev" links to debug HTML pages

TODO:
* Write tests
* Make it look better by right aligning next and left aligning prev
WIP: Report: Strip HTML from notes in table

Come up with a better idea.
WIP: HTML in notes can use the failure and success classes
Packaging: Add run-time dependency on git

For `stbt batch run`, which puts the git version of the test script
in the reports.
report: Add video to the top of the testrun page

It looks great and makes the awesome video recording feature more
visible.
report: Move toward a more data-oriented design

This will make it easier to change to a database backed data-store from
the current directory-backed one, but the reason I'm doing it now is to
make it easier to customise the extra-columns, as will be necessary to add
invisible extra columns which are search-able, but not visible.  This will
be useful so we can search by full git SHA, but only display the git
version description.
stbt batch report: Generate what data we can at `run-time`

The intention is that in the future the `data` member of `Run` will be
stored in a database.  The intention is that the database stores *facts*
and we wish to perform interpretation on these facts at run-time where
possible.  In this case we move the interpretation of what to do with each
of the files until template rendering time.
report: More data-oriented refactoring
report: Refactor: Declare columns in the code rather than in the template

This gives us more flexibility to customise the columns in the report
as is required to do custom formatting on extra-columns.
WIP: report: Allow customising the columns
stbt batch report: Don't HTML escape files we read

Jinja will escape them when writing HTML out and we don't want them
double escaped.  We also want the ability to include HTML in notes.