SafeRun: Move from cros_test_lib to cros_build_lib

In preparation for wider usage in the codebase.

BUG=None
TEST=Unit tests.

Change-Id: I5c2ca368739562e0f2dbc0389230946f17e29661
Reviewed-on: https://gerrit.chromium.org/gerrit/42328
Commit-Queue: Ryan Cui <rcui@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
Make cros asan bots use 2gb rootfs.

This is needed because we are going to use more debug info inside the
image (see the bug) and use more images with asan (shill is in queue).

BUG=chromium-os:38088
TEST=cbuildbot x86-generic-asan amd64-generic-asan --remote -g I6f497214 [-g I07d31d8b]
     buildbot/cbuildbot_commands_unittest.py
     buildbot/cbuildbot_config_unittest.py
     buildbot/cbuildbot_stages_unittest.py

Change-Id: I07d31d8b3015df88e7725188da85082dd5e53647
Reviewed-on: https://gerrit.chromium.org/gerrit/41652
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Denis Glotov <glotov@chromium.org>
Commit-Queue: Denis Glotov <glotov@chromium.org>
Update arguments for make_netboot.sh

make_netboot now takes --image_dir argument instead of --image. Update
it here.

BUG=chrome-os-partner:17368
TEST=Trybot success.
CQ-DEPEND=CL:42095

Change-Id: I5234695ea10106915f34f881df5a0c56c804e2e4
Signed-off-by: Vic Yang <victoryang@chromium.org>
Reviewed-on: https://gerrit.chromium.org/gerrit/42108
Reviewed-by: Chris Sosa <sosa@chromium.org>
deploy_chrome: Default to deploying everything we can find.

The default mode is now to deploy everything we know of that exists.
This mode suits the development workflow better, since developers don't
need to specify extra flags.

BUG=None
TEST=unit tests.  Ran with --gs-path and --staging-only.

Change-Id: I8d54daed6c084aee4822d2ee19a4d443f48bee87
Reviewed-on: https://gerrit.chromium.org/gerrit/42561
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
Revert "[chromite] Add a board whitelist for --hwtest tryjobs"

This reverts commit 15e519f7bd5fd0ac07564fb5eca3064831a9b564

Change-Id: I8a19d30651ba2380a5f1e4af4f47ff343e76a066
Reviewed-on: https://gerrit.chromium.org/gerrit/42654
Reviewed-by: Ryan Cui <rcui@chromium.org>
Reviewed-by: Simran Basi <sbasi@chromium.org>
Tested-by: Simran Basi <sbasi@chromium.org>
osutils: Add SourceEnvironment function.

SourceEnvironment allows you to extract the environment variables set by
a sh script.

Used by cros_chrome_sdk.py - see
https://gerrit.chromium.org/gerrit/#/c/39744/.

BUG=None
TEST=Unit tests.

Change-Id: I5633e5870d1fbafecdd62dfdf71506731576575f
Reviewed-on: https://gerrit.chromium.org/gerrit/42327
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
Un-revert and fix "[chromite] Add a board whitelist for --hwtest tryjobs"

Fixes original commit, by removing the check for people attempting to run
a local build with hwtest.

This reverts commit eb3c33199082030e1a6e8ccb8b3db7a052bfe381.
Original change id: I9c545de4255bc9039fd00a3b8dec1df592bdd278

Change-Id: If4ad0ac462bb271ba6a17f65962309afe5f70aa0
BUG=chromium-os:38642
TEST=Kicked off try job with cbuildbot --remote --hwtest lumpy-release
Reviewed-on: https://gerrit.chromium.org/gerrit/42680
Reviewed-by: Chris Sosa <sosa@chromium.org>
Commit-Queue: Aviv Keshet <akeshet@chromium.org>
Reviewed-by: Aviv Keshet <akeshet@chromium.org>
Tested-by: Aviv Keshet <akeshet@chromium.org>
osutils.SourceEnvironment: simplify a bit

This avoids writing to a temp file, and supports extracting set-but-empty
variables from the environment.  It also supports reading from files that
just echo to stdout.

While we're here, we need to fix a minor bug in LoadKeyValueFile dealing
with 'VAR=' lines.

TEST=`./osutils_unittest.py` passes
TEST=`./cros_build_lib_unittest.py` passes

Change-Id: I549221d0be2edd636cf11551066202a0de0581bd
Reviewed-on: https://gerrit.chromium.org/gerrit/42709
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
manifest_version: convert to osutils.SourceEnvironment

No need to implement the file parser ourselves when we have a
helper to do it now.

BUG=None
TEST=`./manifest_version_unittest.py` passes

Change-Id: I2fbe995586f9679a849ded82744167958689f5d5
Reviewed-on: https://gerrit.chromium.org/gerrit/42711
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Revert "manifest_version: convert to osutils.SourceEnvironment"

CL:42711 might have caused chromium-os:38745 . I am reverting
the CL here to see if this CL caused the problem.

BUG=chromium-os:38745
TEST=none

Change-Id: I5332d185877474ca3165d7c4b0109c4de3eb7733
Reviewed-on: https://gerrit.chromium.org/gerrit/42888
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Add 'cros chrome-sdk' command.

Pulls down SDK components for building and testing Chrome for ChromeOS,
sets up the environment for building Chrome, and runs a command in the
environment, starting a bash session if no command is specified.

For the version of ChromeOS specified (latest build if unspecified), the
command will download and cache:

  1. A dev image of the specified ChromeOS version.
  2. Generated scripts that are used to operate on the image.
  3. A sysroot containing build dependencies of Chrome.
  4. The required cross-compiler toolchain.
  5. The Chrome ebuild environment.

BUG=chromium-os:37360
TEST=unit tests.

Change-Id: I718ac1687c9d38aa62c62ce74ceceaa9f7c8c38f
Reviewed-on: https://gerrit.chromium.org/gerrit/39744
Commit-Queue: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Use the Netflix helper plugin on Daisy.

BUG=chrome-os-partner:5255
TEST=Remote trybot run of daisy-release; cbuildbot_config_unittest.py.

Change-Id: If6d7ad905130d515177af90d9f1da55cb09fa0dd
Reviewed-on: https://gerrit.chromium.org/gerrit/42880
Reviewed-by: David Dorwin <ddorwin@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Xiaohan Wang <xhwang@chromium.org>
Reviewed-by: Xiaohan Wang <xhwang@chromium.org>
Tested-by: Xiaohan Wang <xhwang@chromium.org>
Move uploading of VMTest artifacts to VMTest stage.

This move is required in order to add information about Chrome crashes
into the VMTest stage.

This is a change that Chris Sosa and I have wanted to do for a long time but
never got around to implementing.

Complex communication between stages can make our lives miserable because it
makes for more complicated code and harder to read logs.

Some advantages of this change:

1. Archive stage now no longer waits for the VMTest stage. This means that
   it's now easier to understand why the Archive stage is slow, if it's slow.

2. Archive stage is now a lot simpler in general. The previous, complex
   interaction between VMTest and Archive often lead to bugs in corner cases,
   such as when the VMTest stage is skipped.

Besides the above, I've also cleaned up the output and tests a bit, so that
the VMTest stage doesn't get overloaded with a ton of unnecessary output.
Since this involved removing the logging statements whenever a gsutil command
was run, I also added a timeout here so that we'd always know what happened if
a gsutil command hangs for >60 minutes.

BUG=chromium-os:38586
TEST=Unit tests and remote trybot runs, including failure cases.
TEST=Verified VMTest output is the same in failure and success cases.

Change-Id: I451717539c3794446656aac8bd30329fea53454c
Reviewed-on: https://gerrit.chromium.org/gerrit/42903
Commit-Queue: David James <davidjames@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Add links to crashes in VMTest stage.

BUG=chromium-os:38586
TEST=Unit tests and remote trybot runs, including failure cases.

Change-Id: I2bcca8730cf442486469ac87cdc0366c42ab8a00
Reviewed-on: https://gerrit.chromium.org/gerrit/42964
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Add sys-kernel/puppy-kernel to the blacklist

We have a public puppy overlay, but the kernel source is hosted on a
different repository.  For now, prevent marking as stable so we don't
mess up the paladin builder.

BUG=chrome-os-partner:17778
TEST=None

Change-Id: I4d8b50e4ede96bc664a75771c704fa79dd1f2910
Reviewed-on: https://gerrit.chromium.org/gerrit/43056
Tested-by: Andrew Chew <achew@nvidia.com>
Reviewed-by: Katie Roberts-Hoffman <katierh@chromium.org>
Reviewed-by: Doug Anderson <dianders@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Andrew Chew <achew@nvidia.com>
Remove the --gerrit-chrome option.

The Gerrit Chrome workflow is scheduled to be deprecated, so there is no need
to support it in buildbot. I'm removing this option to help reduce maintenance
costs.

BUG=chromium-os:36324
TEST=Run all unit tests.

Change-Id: I87923acb5d11b3de307feb9ae64b1abcea8175f1
Reviewed-on: https://gerrit.chromium.org/gerrit/43076
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
Tested-by: David James <davidjames@chromium.org>
Move Chrome USE flags to constants.

BUG=chromium-os:36324
TEST=Run all unit tests.

Change-Id: I65e1a2d086af718d110b109402774a4379c4d826
Reviewed-on: https://gerrit.chromium.org/gerrit/43077
Tested-by: David James <davidjames@chromium.org>
Reviewed-by: Ryan Cui <rcui@chromium.org>
Commit-Queue: David James <davidjames@chromium.org>
cbuildbot: update dev-install path

In CL:42992, we moved the dev-install path from /etc to /usr/share.
Update cbuildbot to look at the new location.

BUG=chromium-os:38822
TEST=`./cbuildbot_commands_unittest.py` passes

Change-Id: I3e249246de22eaa371c3ab7a590a34ac39659773
Reviewed-on: https://gerrit.chromium.org/gerrit/43086
Commit-Queue: Mike Frysinger <vapier@chromium.org>
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Add <chrome>/src/third_party dir to PYTHONPATH for Chrome checkouts.

If chromite is living inside the Chrome checkout under
<chrome_root>/src/third_party/chromite, its dependencies will be
checked out to <chrome_root>/src/third_party instead of the normal
chromite/third_party location due to git-submodule limitations (a
submodule cannot be contained inside another submodule's workspace),
so we want to add that to the search path.

BUG=None
TEST=Local

Change-Id: I10a12bcddc88e509c1c7015a95c54d578fb8b122
Reviewed-on: https://gerrit.chromium.org/gerrit/43066
Reviewed-by: Mike Frysinger <vapier@chromium.org>
Commit-Queue: Ryan Cui <rcui@chromium.org>
Tested-by: Ryan Cui <rcui@chromium.org>
cbuildbot: fix call to upload_prebuilts

The current handwritten regex does not properly handle all atoms.
Specifically, it fails with packages whose PV is a single integer:
  chromeos-base/libchrome-125070-r6
  net-misc/iputils-20100418-r3

Rewrite it to use the existing SplitCPV helper in portage_utilities.

BUG=chromium-os:38296
TEST=`./buildbot/cbuildbot_commands_unittest.py` passes
TEST=`cbuildbot daisy-release` worked (reviewed DevInstallerPrebuilts logs)

Change-Id: I476c14368e024f3a8bea1426280d66064ff10ecd
Reviewed-on: https://gerrit.chromium.org/gerrit/43094
Tested-by: Mike Frysinger <vapier@chromium.org>
Reviewed-by: David James <davidjames@chromium.org>
Commit-Queue: Mike Frysinger <vapier@chromium.org>