Catch and propagate up SystemExit errors.

In the case where a child process raises a SystemExit or KeyboardInterrupt
exception, it is important that it gets pushed properly onto the queue so
that the parent process doesn't deadlock waiting for the input on the queue.

BUG=chromium-os:37519
TEST=Run example trybot run with deadlock caused by cros_build_lib.Die(),
     verify this CL fixes the deadlock.
Change-Id: Ibeb899c49d6a87eecda5d3a30f7129f4bf5995e9
Reviewed-on: https://gerrit.chromium.org/gerrit/40293
Reviewed-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Switch perf builders to use chromeos_lkgm instead of lkgm.

Also includes a switch to wait longer for chromium tree to open as its
most important to get the lkgm open.

BUG=None
TEST=Unittests + pylint on changed files.

Change-Id: I593f65c8847969c2b30867e20d3deb0b85357835
Reviewed-on: https://gerrit.chromium.org/gerrit/40231
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>
Fix more parallel deadlocks and add unit tests.

When items are pushed to a queue that is read by background processes, the
background processes must read all items off the queue, or the program may
deadlock. Currently, RunTasksInProcessPool catches Exception, but not
BaseException, so there is no guarantee that it will read all the items off
of the queue.

I verified that this problem can cause hangs in practice by
adding a sys.exit() to the right place in cros_list_buildbot_crashes.

This CL makes the following small changes:
  1. Catch BaseException instead of Exception in _TaskRunner. This ensures
     that _TaskRunner still reads all the items off of the queue in the case
     where a task exits or receives a KeyboardInterrupt.
  2. Grab fileno() from sys.__stderr__ and sys.__stdout__ instead of
     sys.stderr and sys.stdout. Since sys.stderr and sys.stdout may not be
     real files, it's unsafe to use 'fileno' on them. This makes unit testing
     of parallel tests using OutputCapturer() possible.
  3. Don't print tracebacks in RunParallelSteps(). Printing tracebacks here is
     unhelpful because we already include the traceback in the exception that is
     raised. If the caller wants to print the traceback, it can, and in fact
     already does this in all cases. Printing the traceback twice just leads to
     confusing output.
  4. Add more unit tests for testing the case where KeyboardInterrupt and
     SystemExit exceptions are thrown.

BUG=chromium-os:37519
TEST=Unit tests. Trybot runs with successful and failing stages.

Change-Id: I30889e8f85a09a106a66f9c878323aba41c39bf1
Reviewed-on: https://gerrit.chromium.org/gerrit/40295
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Fix broken sanity check.

Fixes the following Exception:
TypeError: coercing to Unicode: need string or buffer, list found

BUG=chromium-os:32407, chromium-os:37471
TEST=Verify code actually works now with a trybot run.

Change-Id: I7155e0761d901669b60e029e4422da35cb312207
Reviewed-on: https://gerrit.chromium.org/gerrit/40298
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Make the bot red if any asan_log found.

BUG=chromium-os:36140
TEST=cbuildbot x86-generic-asan amd64-generic-asan --remote -g ...

Change-Id: I6dbb82e463701d70d832c3cbd933fdb88024be7e
Reviewed-on: https://gerrit.chromium.org/gerrit/40211
Tested-by: Denis Glotov <glotov@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Denis Glotov <glotov@chromium.org>
Update IsTreeOpen to log when tree is closed.

Right now, IsTreeOpen doesn't log anything when the tree is closed, so
buildbot may think that the buildbot isn't doing anything and kill the
builder due to inactivity. Print some helpful status messages so that
it's clearer what the buildbot is doing.

BUG=none
TEST=Example cbuildbot run without --debug. Unit tests.

Change-Id: Ie815d1fb620bbbcb76bdaf253e9c04e19f72bbe3
Reviewed-on: https://gerrit.chromium.org/gerrit/40303
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Force a new process group for our cleanup pathways to avoid SIGKILL.

Do this since buildbot now likes to blast the full process group,
and we have processes that explicitly break out of our process group
(I'm looking at you mr. vmtest kvm process).

Additionally, ensure stdin/stdout/stderr are flushed if at all
possible.

BUG=None
TEST=kill -9'ing cbuildbot.  Oddly therapeutic.

Change-Id: Ie443b4cd48aea6ced2b46be7876c163734b4e366
Reviewed-on: https://gerrit.chromium.org/gerrit/40431
Tested-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Brian Harring <ferringb@chromium.org>
upload_prebuilts: change datestamp format

I find the US date format to be ugly and ambiguous.  It also makes sorting
a huge pita.  So let's fix it.

Using this date as an example:
	Thu Dec 13 07:16:07 UTC 2012
The old code would produce:
	13.12.12.071607
The new code produces:
	2012.12.13.071607

The reason for using %Y over %y is to avoid possible collisions.
In the old format: 13 Dec 2012 00:00:00 -> 13.12.12.000000
In the new format: 12 Dec 2013 00:00:00 -> 13.12.12.000000
Using a 4 digit date avoids that issue.

This shouldn't cause a problem in practice as anytime this version string
has appeared, it's been dumped into a variable as an opaque blob.  That
means no code is parsing it to figure out the date or to infer path info
from it.  Replacing one arbitrary string with another makes no difference
to that code.

As an example, this is being used in binhost URLs:
$ cd src/third_party/chromiumos-overlay/chromeos/binhost/host/
$ grep . *.conf
amd64-LATEST_RELEASE_CHROME_BINHOST.conf:LATEST_RELEASE_CHROME_BINHOST=http://commondatastorage.googleapis.com/chromeos-prebuilt/host/amd64/chrome-27.05.11.033811/packages/
amd64-TOT_CHROME_BINHOST.conf:TOT_CHROME_BINHOST="http://commondatastorage.googleapis.com/chromeos-prebuilt/host/amd64/chrome-26.05.11.214444/packages/"
sdk_version.conf:SDK_LATEST_VERSION="12.12.12.180828"
sdk_version.conf:BOOTSTRAP_LATEST_VERSION="2010.03.09"

Other code simply takes those values as-is and plugs them in to other
places.  Changing it should be safe.

BUG=None
TEST=buildbot/run_tests passes

Change-Id: I7dc4269a89ac7a0d5e984ef7a5be5d12b7430e1c
Reviewed-on: https://gerrit.chromium.org/gerrit/39642
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Reviewed-by: Brian Harring <ferringb@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
GS:// ; Convert the atomic creation code to use generation.

BUG=chromium-os:37718
TEST=manual validation of the behaviour

Change-Id: I8c7fc099fc91c760eb2083dadc607ce4e095c28c
Reviewed-on: https://gerrit.chromium.org/gerrit/40928
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
Force gs.py to use generation rather than sequence-match.

Currently no consuming code has been found sans unittests so
this is reasonably safe.  For existing usage where it was
just a fail-if-exists check, the change restores functionality;
for any sequence tracking code (which I've yet to find sans locks
that don't use this pathway) it will break; it was broken via
the referenced bug anyways, thus is decided to move forward with.

BUG=chromium-os:37718
TEST=validation of the header, but nothing beyond that; TBR.

Change-Id: Ic1d42b0e9dca1be60105f86aa0eaaed6020e0f40
Reviewed-on: https://gerrit.chromium.org/gerrit/40929
Reviewed-by: Brian Harring <ferringb@chromium.org>
Tested-by: Brian Harring <ferringb@chromium.org>
deploy_chrome: run rsync as root.

Some files in the staging dir don't have read access for all users, so
rsync fails to open them.  Solutions is to run rsync as root.

BUG=None
TEST=Deployed to local device.  Unit tests.

Change-Id: I1f69b13cd3bfe85c10e2a127081daa5b54bf32bb
Reviewed-on: https://gerrit.chromium.org/gerrit/40476
Commit-Queue: Ryan Cui <rcui@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
gclient: use cros_build_lib.GetHostName

Minor clean up.

BUG=None
TEST=`./buildbot/run_tests` passed

Change-Id: I62c4938bc6d0c5d9834c034c74834425e789f008
Reviewed-on: https://gerrit.chromium.org/gerrit/40598
Reviewed-by: Peter Mayo <petermayo@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
lddtree: use rpaths from main ELF to locate dependencies

Since the main ELF's rpaths changes which libs get loaded at runtime,
make sure we take that into consideration when looking for files and
not just the sub lib's rpaths.

BUG=chromium-os:36453
TEST=`lddtree /usr/bin/armv7a-unknown-linux-gnueabi-as` finds binutils libs in
	/usr/x86_64-pc-linux-gnu/armv7a-unknown-linux-gnueabi/lib/ rather than /usr/x86_64-pc-linux-gnu/lib/

Change-Id: I5da657a6fece3d455dff21812d1bad3d449326e2
Reviewed-on: https://gerrit.chromium.org/gerrit/40504
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
cros_build_lib: LoadKeyValueFile: new helper func for reading key=value files

This takes existing code in cros_sdk and generalizes it a bit so it can be
used elsewhere (parsing general shell config files).

Then we gut similar logic in layout.conf parsing and use this new func.

BUG=chromium-os:36453
TEST=`./lib/cros_build_lib_unittest.py` passes
TEST=`cros_sdk --chroot=new_chroot` works
TEST=`cbuildbot daisy-paladin` shows egencache running for right repos

Change-Id: I133e188fc983d4cc2d1b0cfccf48a80cc76a8383
Reviewed-on: https://gerrit.chromium.org/gerrit/40603
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Add a config for depthcharge based firmware.

Depthcharge is a bootloader that replaces U-Boot on x86 based systems.

BUG=chrome-os-partner:17147
TEST=Ran cbuildbot_config_unittest.py.

Change-Id: Ida9c00e09fe25437e93f5715803b2136ce913494
Signed-off-by: Gabe Black <gabeblack@google.com>
Reviewed-on: https://gerrit.chromium.org/gerrit/41062
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: Gabe Black <gabeblack@chromium.org>
Commit-Queue: Brian Harring <ferringb@chromium.org>
Fix the list of boards used for the depthcharge config.

My previous commit which added a depthcharge config accidentally used the main
firmware board list instead of the depthcharge specific one.

BUG=chrome-os-partner:17147
TEST=Ran cbuildbot_config_unittest.py.

Change-Id: I1e05402afc187599a88155263b6d7009707666e7
Signed-off-by: Gabe Black <gabeblack@google.com>
Reviewed-on: https://gerrit.chromium.org/gerrit/41112
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Gabe Black <gabeblack@chromium.org>
Tested-by: Gabe Black <gabeblack@chromium.org>
Turn off dev_installer_prebuilts in firmware builders.

Fixes the following error:
http://chromegw/i/chromiumos.tryserver/builders/link-depthcharge-firmware/builds/1

BUG=chrome-os-partner:17147
TEST=Trybot run.

Change-Id: I3070602dc6d7e7576a97737d6ecdd6a467986d03
Reviewed-on: https://gerrit.chromium.org/gerrit/41127
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Tested-by: Gabe Black <gabeblack@chromium.org>
Define reviewers outside of conditional scope.

Fixes bug where reviewers isn't set properly.

BUG=chromium-os:37852
TEST=Ran it/pylint

Change-Id: I275d3e3eeeb11caee7d7b78507c7a9478547f6c9
Reviewed-on: https://gerrit.chromium.org/gerrit/41222
Tested-by: Chris Sosa <sosa@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Chris Sosa <sosa@chromium.org>
cros_setup_toolchains: add support for building redistributable cross-compiler packages

This adds a new flag to create a tarball for each cross-compiler.  This
tarball includes binutils, gcc, and gdb.  We omit glibc and the kernel
headers as the expectation is these will be used with a sysroot.  That
will include the glibc/kernel headers as needed.

Once you unpack the tarball, you just have to add the toplevel bin/
directory to your path.  All tools will be found in there.

To avoid the conflicting glibc issue, we include every library that the
toolchain needs in order to execute.  We then wrap every executable ELF
with a shell script.  This script will run the local ldso directly as
well as tell it where to find the libraries.

BUG=chromium-os:36453
TEST=`cros_setup_toolchains --create-packages` builds pkgs for each target
TEST=unpacked tarballs can be used in gPrecise (outside of the chroot)
CQ-DEPEND=CL:37878

Change-Id: I8e88661f6bfcba6cca5aad629b4acfcf7b7c1b98
Reviewed-on: https://gerrit.chromium.org/gerrit/38269
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
cbuildbot_config: add x32 profiles

BUG=chromium-os:36866
TEST=`cbuildbot_config_unittests.py` passed

Change-Id: Iefbae215e01be13ff469fe3fe50560da93c65e88
Reviewed-on: https://gerrit.chromium.org/gerrit/41261
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>