bitbake: toasterui: refactor log saving and save out-of-build errors

We refactor log saving to go through only one code path.

All logs that happened outside the build (i.e. before build
starting) now will be logged to either toaster_ui.log if the
build command ran in interactive mode, or to the build request
errors if the command ran in managed mode.

This enables proper display of error logs in project page.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: toaster: fix application discovery in settings.py

We fix application discovery, tuples don't have .append().

Other minor fixes.

(Bitbake rev: a6f18aac3e6bb448d89a3425a2f756c6514ee595)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: toaster: update orm models for layerindex compatibility

We add a ToasterSettings table that will keep
installation-wide settings.

We update the models for the layer-related data storage to
make them compatible with the layerindex application API.

We add a LayerSource class that can update local data from
a LayerIndex-like compatible API.

Adding a command line option to perform information update
from all upstream layer sources.

Fair warning - there is no backward migration from 0013.

(Bitbake rev: 89e13579e1b44b738f10fadec8454aa0e6f073af)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: toaster: enable admin interface on select models

We enable the admin interface in Toaster.

We add admin models for Build Environments (where the
sysadmin can configure where the builds take place)
and for Layer Sources (marking the upstream provider
for layer information).

The admin interface and associated data are enabled
only for the MANAGED version.

(Bitbake rev: 6618613c9210fb44d36d90f5f2404b435f10dfc8)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: toaster: create project section navigation structure

We create the navigation structure for the project section.
This includes adding URLs for configuration, builds, all layers,
layer details and all targets pages.

Changes to existing pages to exemplify navigation links.

(Bitbake rev: 6f0cb9d106129eb496a4c009d95b0727378e97c1)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: toaster: add all layers page

We add a page where the user can browse
and import layers from all the layers known
by Toaster.

[YOCTO #6590]

(Bitbake rev: 59f4a9750a6c4f5360a91e3a4d1c03ceb42da086)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: toaster: update checksettings command for auto-detection

We enhance the checksettings command to try to automatically
detect settings for running on localhost.

The checksettings will look for a nearby poky layer source,
for a nearby build directory, and will try to import settings
from "toasterconf.json" files found in the local layer.

On new configuration, it will also perform updates from the
layer source.

(Bitbake rev: 2aab77dfccb538e2b09829841ea6c464d40cafb1)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: toaster: update the bldcontrol to the new orm models

We update the build controller application to make proper
use of the bitbake specification in project settings.

Added heuristic to detect when the meta* layers and bitbake
are checked out from Yocto Project poky, and use a single
git checkout.

Building without a proper oe-init-build-env is not yet supported.

(Bitbake rev: 9eafe14956013f5af39b68fc93e1b03e7ea1f5c2)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: toatergui: update pages to match project models

We update pages and queries to match the new project
models.

(Bitbake rev: 8f47ec259106da714260c7388cee75a6c1f4622d)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: buildinfohelper: BuildRequest project file update soft linked

(Bitbake rev: 93887dadd5ee35557d320e96059c466d2e541065)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: toaster: create Build methods for calculating progress and ETA

We move the code to calculate build progress as percent
and the ETA of the build to the model, so that they can be
reused across different pages.

(Bitbake rev: c2ced09e7ea4a1762d2788bb12a761734d20fd8e)

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lttng-tools: fix ptest execution failure

Ptest will hang when execute "./run.sh fast_regression" and lttng background
daemon can't be killed normally. This patch fixes this issue and use ${PTEST_PATH}
instead of long path.

(From OE-Core rev: 0a90616b09290e4651aafe1186229d861392e89f)

Signed-off-by: Chong Lu <Chong.Lu@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
buildtools-tarball: include nativesdk-ca-certificates

nativesdk-ca-certificates is needed in order to support oe/yocto builds with
buildtools-tarball on old hosts, as we provide libcurl, and that needs to be
able to find the certs, and there's no standard path or bundle path, so we
can't rely on the host.

(From OE-Core rev: 057954bcc4baf9194968169591537cd7584f8bcb)

Signed-off-by: Fahad Usman <fahad.usman@gmail.com>
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
buildtools-tarball: export GIT_SSL_CAINFO

export GIT_SSL_CAINFO so git can find the certs

(From OE-Core rev: 260ea7b85ecacf2fcd4d09fa2f602d3e4368ef69)

Signed-off-by: Fahad Usman <fahad.usman@gmail.com>
Signed-off-by: Christopher Larson <kergoth@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
classes/cpan_build.bbclass: add EXTRA_CPAN_BUILD_FLAGS

Some packages have additional configuration options that are currently
not accessible through cpan_build.bbclass. Similar to cpan.bbclass, add
EXTRA_CPAN_BUILD_FLAGS to pass the optional parameters to perl Build.PL.

(From OE-Core rev: bef6c3f4ebcd19ea3845aab27a165c370102b62c)

Signed-off-by: Tim Orling <TicoTimo@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/runtime: Automatic test for parsing the logs on a machine and search for certain error keywords.

This adds a common new qa test for general processing of log files. One
significant improvement is machine dependent ignore filters.

This can be used to replace several weaker individual QA tests
that are currently used.

(From OE-Core rev: a14d076a401397b6773d5d1b99e49126261f1eb4)

Signed-off-by: Lucian Musat <georgex.l.musat@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa/pasrselogs: Improve the machine/string whitelist

Currently the whitelist is imcomplete, inaccurate and suffers duplication.

These changes:
 * Add common groups of errors
 * Change to make the default whitelist clear
 * Correctly (or at least better) escape the regexp expressions
 * Add in missing machines to allow builds on the autobuilder to suceed

(From OE-Core rev: 620aa5f9022335a9166b4d47bdcdce611ff5466a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
testimage: Add parselogs to the default tests for sato images

(From OE-Core rev: d373d4332a03eb71c1112b8dd51052e56ca4fa59)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
oeqa: xorg log test is being replaced by parselogs so remove

Now we have the parselogs test, this one can be removed.

(From OE-Core rev: 7977a3c28677d9a248059b0be230f345227e798a)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/oe/utils: Make multiprocess_exec() return anything

The variable "results" was accidentally used for multiple different
things at the same time, which unintentionally discarded anything that
was supposed to be returned from the function...

(From OE-Core rev: abf4eb613eba0892a5f240de7aa3a9a1b2879354)

Signed-off-by: Peter Kjellerstedt <peter.kjellerstedt@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>