Initial ugly commit.

Definitely some enhancements can be done. I think I have everything
needed but I have not been able to test this yet. If this needs to
get done before I am back feel free to work on it.

Completely untested and probably a few mistakes somewhere...

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
Distro kernel bug-fixes.

Fixed some things that were being done incorrectly.

Some distro kernels have no debug so added | true when disabling
kdb. Also changed what was skipping kernels if non-ubuntu to also
schedule kernel install if a distro kernel.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
Add some debug logging.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
Added docstrings.  Cleaned up code (broke up long lines, removed unused
variable references, pep8 formatted most of the code (one set of long lines
remains), and changed some variable and method names to conform to pylint
standards).

Fixes: 6530
Implement a watchdog for queued jobs

This continually posts the run's status to the results server, if
configured, at an interval defaulting to 600 seconds.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
Merge pull request #157 from ceph/wip-watchdog

Implement a watchdog for queued jobs
nuke: fix sync before reboot timeout

If you do 'timeout 5 sync' and sync hangs, timeout will block trying to
kill it.

Instead, just background sync, wait a few seconds, and reboot.  This means
we wait a few seconds even if sync returns immediately, but who cares!

Signed-off-by: Sage Weil <sage@inktank.com>
ceph.conf: default to 2x

A bunch of our tests rely on this; they need to be fixed
before we can run at 3x.

Signed-off-by: Sage Weil <sage@inktank.com>
Added additional comments.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
Remove unused variable.

Signed-off-by: Sandon Van Ness <sandon@inktank.com>
nuke: ignore exceptions while issuing reboot command

I'm seeing failed tasks (and nuke) leak machines.  It looks like we are
getting an exception on the '... reboot -f -n' command when we should be
ignoring it and waiting for the machine to restart.

For example:
   http://qa-proxy.ceph.com/teuthology/sage-2013-12-08_19:25:06-rados:thrash-wip-tier-foo-basic-plana/136321/teuthology.log

Signed-off-by: Sage Weil <sage@inktank.com>
Merge pull request #158 from ceph/wip-nuke

make nuke behave
Auto-restart

If /tmp/teuthology-restart-workers is newer than the running process,
restart.

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
Merge pull request #151 from ceph/wip-distro-kernel

Wip distro kernel
Add missing req: psutil
Make -a optional

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
Create a DateTime object from the timestamp

Signed-off-by: Zack Cerza <zack.cerza@inktank.com>
ceph.conf: put 2x command in [global]

so that osdmaptool sees it.

Signed-off-by: Sage Weil <sage@inktank.com>
rados: allow existing pool(s) to be used

Signed-off-by: Sage Weil <sage@inktank.com>
Merge pull request #159 from ceph/wip-cache

rados: allow existing pool(s) to be used