ConfHandler: Improve regexp to fix mis-parsing of += and no whitespace

If you have:

FOO = "a"
FOO += "b"
FOO+= "c"

The expected result is "a b c" however we were seeing "a b" with the FOO+
variable being assigned the value "c". This isn't the expected result.

We need to make the name part of the variale non-greedy so that any + character
becomes part of the operator. This patch does that. I compared the configuration
in OE-Core before and after the change and only the test case changed.

[YOCTO #3834]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove whitelisted vars from non-task deps

Though the value of variables in the BB_BASEHASH_WHITELIST is kept out of the
checksums, dependency on them is not, at least for variables and non-task
functions. In the code, the whitelist is removed from the overall task dep
list, but not the individual variable deps. The result of this is that
functions like sysroot_stage_all and oe_runmake end up with whitelisted
variables like TERM listed in their dependencies, which means that doing
a 'unset TERM' before building will result in all checksums for tasks that
depend on those changing, and shared state reuse not behaving correctly.

This is only really a potential issue for variables from the environment, as
it's the existance/removal of the variable that's an issue, not its value, and
the other whitelisted variables are set in our metadata. This which means in
practical terms the only cases where this is likely to be an issue are in
environments where one of the following are unset: TERM, LOGNAME, HOME, USER,
PWD, SHELL. This may seem like an unlikely circumstance, but is in fact a real
issue for those of us using autobuilders. Jenkins does not set TERM when
executing shell, which means shared state archives produced by your jenkins
server would not be fully reused by an actual user.

Fixed by removing the whitelisted elements from the individual variable deps,
not just the accumulated result.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
codeparser: Track appendVar and prependVar calls as we do for getVar

We need to track appendVar and prependVar calls just as we do for getVar in order
to ensure we're not missing variable dependencies.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
codeparser: Track bb.utils.contains usage

The bb.utils.contains function usage is getting increasingly used in the metadata
but isn't handled automatically by the python dependency tracking code. This patch
changes that and also adds the "OE" names for the functions.

Whilst there are reasons this is a bad idea, its likely outweighed by the
shear number of these references and the current holes in dependency information
which we're now relying heavily upon.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
wget: Only use -O option when not spidering

If we pass the -O option whilst spidering, empty files are created which is not
desired. We also need to ensure any subdirectories are created when using the
downloadfilename parameter.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
tests/codeparser: Hack around circular inclusion problem

There is a circular dependency issue where bb.data can't directly depend on
bb.parse. The tests were failing due to this issue which is misleading. This
patch hacks around it for now. I'd rather that than not running tests at all.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ConfHandler: Use re.X to make long regexp more readable

The __config_regexp__ in ConfHandler is quite long, and using re.X to
break the expression onto several lines make it a bit easier to read.

Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Ensure expansions happen in URL parameters in uri_replace

For example, this allows us to specify downloadfilename=PATH in sstate
mirror urls to improve directory structures in DL_DIR when using sstate.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake & hob: reparse config files, when sanity check updates bblayers.conf

I've sent a patch to poky, with some changes in sanity.bbclass. When bblayers.conf
is updated, it updates also a variable to know when to reparse configuration files.
This patch contains the changes to reparse conf files, after a sanity check which
changes bblayers.conf.

[YOCTO #3213]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Improve lock/done stamp file paths for local files

Currently stamps end up at the top level of DL_DIR even if the files
themselves are in a directory structure. This patch preserves path
components allowing the top level directory to be less populated
which is an advantage for large sstate caches.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
monitordisk.py: disable the inode checking for some fs

There is an error when use disk monitor on btrfs:
WARNING: The free inode of rootfs is running low (0.000K left)
ERROR: Immediately abort since the disk space monitor action is "ABORT"!

This is beucase some fs formats' statvfs.f_files (inodes) is zero, thus
the statvfs.f_favail (free inodes) is zero, too, this a feature of the
fs, we disable the inode checking for such a fs.

[YOCTO #3609]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob: no sanity checks re-run

Hob was hanging when the user tried a rebuild a new image, because
of sanity checks. It should not re-run sanity checks.

[YOCTO #3829]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob: remove parsing warnings when machine is changed

When the machine is changed, the warnings list is emptied and the warnings
bar is hidden.

[YOCTO #3830]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-layers: make show-cross-depends avoid long path

The "bitbake-layers show-cross-depends" print the absolute path in the
past, now it will print the relative path (relative to the layer dir) by
default, and add the "-f" option to make it print the absolute path.

For example:

$ bitbake-layers show-cross-depends
[snip]
meta-intel/meta-jasperforest/conf/machine/jasperforest.conf requires meta/conf/machine/include/ia32-base.inc
[snip]

$ bitbake-layers show-cross-depends -f
[snip]
/path/to/poky/meta-intel/meta-jasperforest/conf/machine/jasperforest.conf requires /path/to/poky/meta/conf/machine/include/ia32-base.inc
[snip]

[YOCTO #3824]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
knotty.py: fix unknown event bb.event.DiskFull

There is an error:

ERROR: No new tasks can be excuted since the disk space monitor action
is "STOPTASKS"!
ERROR: Unknown event: <bb.event.DiskFull object at 0x2ab6310>

This is because we don't handle the event bb.event.DiskFull in
knotty.py, we don't want the knotty to do anything here since we have
done everything in monitordisk.py, so just ignore this event would fix
the problem.

[YOCTO #3523]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob: remove save/load template functionality

The Hob templates functionality, in its current incarnation,
is confusing and has no clear utility.

[YOCTO #3696]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
utils: Use rm -rf in remove()

Whilst shutils.rmtree() is pythonic, its also slow. Its faster to
use rm -rf which makes optimial use of the right syscalls.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Ensure directory for stampfile exists before trying to create it

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
world-image: add extra target

* similar to world target, but collects all PACKAGES
* recipes marked with EXCLUDE_FROM_WORLD are excluded from world as well
  as world-image
* recipes marked with EXCLUDE_FROM_WORLD_IMAGE are excluded completely
  from world-image, sometimes you can build 2 recipes in same sysroot,
  but cannot install them in the same image
* PACKAGES listed in EXCLUDE_PACKAGES_FROM_WORLD_IMAGE are also
  excluded. You can remove e.g. ${PN}-dbg, but keep all other PACKAGES.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>