bitbake/runqueue: Improve the setscene logging

When debugging setscene problems it was found that some extra debug messages were useful.
This patch adds them.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker.py: Add missing BB_WORKERCONTEXT conditionals

Its desireable to run be able to execute bitbake tasks directly, only
loading as much of the cooker is as necessary to do this. This use case
can be identified by the BB_WORKERCONTEXT variable. There were a couple
of places cooker code needed to be conditional on this but wasn't. This
brings in the missing conditionals from poky where those code paths
are used more heavily completing the support in master.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
command.py: Modify needcache value for certain functions

for findConfigFiels() and findFilesMatchingInDir() functions, they
don't need to parse all the bb files, thus setting the needcache
value to be False.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker: user bb.configuration.data to inject events

For buildTargets function, we use bb.configuration.data as parameter
to inject events, since in hob environment, some variables are
modified dynamically and bb.configuration.event_data may out of date.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
command.py: add initCooker API

initCooker is to set the cooker to the initial state with nothing
parsed.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
xmlrpc: Change BitbakeServerInfo init function

Pass host and port to BitbakeServerInfo class instead of the "server"
instance. With this change, GUI can connect with server individually
by host address and port.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker: remove command import in cooker.py

There is no direct use of command in cooker.py, and it is using
bb.command instead. Remove command in the import list.

This fixes a problem of embedded import between command.py and
cooker.py.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker: remove command import in cooker.py

There is no direct use of command in cooker.py, and it is using
bb.command instead. Remove command in the import list.

This fixes a problem of embedded import between command.py and
cooker.py.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: add a new option "--server-only"

Create a new option "--server-only" for bitbake command, which allows
bitbake runs as a server, and let frontend connect the server itself.

"--server-only" should work with "-t xmlrpc", or bitbake will exit.

bitbake --server-only -t xmlrpc will print out the server address and
port information.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
command.py: add parseConfigurationFiles API

The parseConfigurationFiles API calls the related function in
cooker.py to parse config files.

Signed-off-by: Dongxiao Xu <dongxiao.xu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-layers: flatten: allow specifying layers to flatten

You can now optionally specify two or more layers to flatten into the
output, rather than flattening all of the layers in the current
configuration (but this is still the default behaviour if no layers are
specified). Note that this means the output layer may still contain
bbappends where the corresponding recipes are not present in the list of
layers to flatten. There is also a caveat when a layer not being
flattened would be "inbetween" the flattened layers (see the command
help for details.)

Implements feature request in [YOCTO #1564].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-layers: flatten: warn the user if output structure is incorrect

If you flatten layers that have different directory structures you may
not end up with a usable layer in the output directory - some files
won't be picked up by BitBake.

To try to avoid this problem, once flattening has completed, get the
BBFILES entries that correspond to the layer from which the output
layer's conf/layer.conf came from, and check through all of the
.bb/.bbappend files in the output directory to see if any will not be
referred to by BBFILES in the output layer. If any are found, show a
warning to the user.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-layers: close files in apply_append()

It's recommended practice to close files when finished with them and the
code in this function was not doing this.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake/knotty: Show summary of warning/error messages shown

Show a summary count of warning/errors messages shown to the user during the
build and make it clear when an error exit code is being set.

[YOCTO #1540]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: Add BBHandledException exception class

We have a problem knowing when to show the user debug information and
when not to since the code has already shown the user suitable information
about why a failure is occurring.

This patch adds a bb.BBHandledException exception class which can be used
to identify those exceptions which don't need further explanation to
the user.

This patch uses this class for the bb.providers exceptions and ensures the
command handling code correctly filters the exceptions meaning that

"bitbake invalid"

now shows an simple error message and not a python traceback.

[YOCTO #1141 partial]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake/PRservice: Added no_hist mode and export/import.

[YOCTO #1556]
1. Added the package_arch into the index to the DB table. Because the
change in PACKAGE_ARCH will results in different checksum, and it is
better to have seperate PR value domains for differnt PACKAGE_ARCH of
the same pakcage.

2. Changed the PR service to operate in no history mode. In this mode,
the for a given query tuple (version, pkgarch, checksum), the returned
value will be the largest among all the values of the same (version,
pkgarch). This means the PR value returned can NOT be decremented.

3. Added export function. For each (version, pkgarch) tuple, only the
record with the maximum value will be exported.

4. Added import function. The record will only be imported if the
imported value is larger than the value stored in the DB with the same
(version, pkgarch, checksum) tuple.

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: Automatically start local PR service.

[YOCTO #1126]
A local PR service will be started and stopped automatically along
with the bitbake invocation/ternimation.

This local PR service will be started only and if only when the
PRSERV_HOST is set to 'localhost' and PRSERV_PORT is set to '0'.

When started, the sqlite3 database is stored at
"${PERSISTEN_DIR}/prserv.sqlite3" or "${CACHE}/prserv.sqlite3".

Signed-off-by: Lianhao Lu <lianhao.lu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake/knotty: don't count errors as warnings in summary

The count of warnings being shown in the summary at the end was also
including the number of errors.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake/fetch2: improve error formatting for fetcher errors

* The "name" argument to FuncFailed is rarely used as a name in actual
  usage within bitbake, so don't treat it as one in the output.
* Don't print URL for FetchError if it was not specified (i.e. don't
  output "Fetcher failure for URL 'None'")
* Don't include URL in "unable to fetch from any source" message since
  we supply it to FetchError and it will be printed anyway.
* Don't include URL in "checksum failed" message for the same reason

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake/runqueue: avoid "failed" in task summary if nothing did

Seeing the word "failed" alone without reading the whole context has
occasionally triggered an automatic assumption on the part of some users
(myself included) that something has gone wrong, even when this message
is telling you that "0 [tasks] failed". To avoid this let's just say
"all succeeded" in this case instead.

As a bonus this means you can now search the output for "fail" and not
find anything if all went well.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>