bitbake: fetch2: Print the complete SRCREV variable name when INVALID

If a particular SRCREV (say for a particular branch) is missing, the
fetcher will currently just report an obtuse error about the "SRCREV"
being invalid. If there is more information is to be had (say from name,
i.e. branch, and pn) then display that as well.

The new error looks something like this:

ERROR: ExpansionError during parsing /home/dvhart/source/poky/meta/recipes-kernel/linux/linux-yocto_3.4.bb: Failure expanding variable do_patch: ExpansionError: Failure expanding variable SRCPV, expression was ${@bb.fetch2.get_srcrev(d)} which triggered exception FetchError: Fetcher failure for URL: 'git://otcgit.jf.intel.com/dvhart/linux-yocto-minnow-3.4.git;protocol=git;nocheckout=1;branch=standard/minnow,meta,emgd-1.14;name=machine,meta,emgd'. Please set SRCREV_emgd_pn-linux-yocto to a valid value

Note the variable listed as invalid is
"SRCREV_emgd_pn-linux-yocto", making it explicit what is wrong.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Cc: bitbake-devel@lists.openembedded.org
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
compat/utils: Add copy of python multiprocessing pool for pre 2.7.3 issues

python 2.7 shows hangs with issues in its pool implmenetation. Rather than
try and hack around these, add a copy of the working pool implementation
to the compat module from 2.7.3.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker/utils: Drop custom 'interactive' variables list and store environment in BB_ORIGENV instead

Maintining hardcoded lists of user environmental variables is hacky,
replace these with the savedenv datastore. Allow access to that
through the BB_ORIGENV variable.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Export upper and lower case environment variables

Applications are inconsistent in their use of upper and lower case proxy
variables. Curl, for example, specifies NO_PROXY (not no_proxy) in the
man page (changed in 2009 [1]). Avoid proxy issues by ensuring both the
upper and lower case versions of each proxy variable is available in the
environment for the fetcher commands.

Add FTPS_PROXY and ftps_proxy to the list as well.

1. http://curl.haxx.se/mail/tracker-2009-04/0012.html

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Remove broken git variables from the environment

The following variables perform no function outside of bitbake:

    GIT_CONFIG
    GIT_PROXY_HOST
    GIT_PROXY_PORT
    GIT_PROXY_IGNORE

GIT_CONFIG only affects the git-config command which is not relevant to
the fetcher. This was previously used with the OE GIT_CORE_CONFIG
variable which would provide a basic git config to use instead of the
user's config. This usage was deprecated by git for over a year now:

http://git.661346.n2.nabble.com/Overriding-gitconfig-using-GIT-CONFIG-td6680977.html

GIT_PROXY_HOST and GIT_PROXY_PORT are not used by git.

GIT_PROXY_IGNORE was an OE construct used to create the custom git
config and had no meaning outside of the OE environment. It is not used
by git.

Remove these variables from the fetcher environment.

Users wishing to configure git to work with a proxy should define the
GIT_PROXY_COMMAND environment variable to use an external script.
NO_PROXY can be used within this script to skip the proxy for certain
hosts.

Signed-off-by: Darren Hart <dvhart@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Update version to 1.17.0

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: Always use separate process for PR Service

Using the threading module interacts badly with multiprocessing used elsewhere
in bitbake under certain machine loads. This was leading to bitbake hanging on
Ctrl+C when the PR Server was being used.

This patch converts it to always use the daemonize code which
then means the threading code isn't required.

[YOCTO #3742]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv/serv.py: Fix logging in daemon mode

In deamon mode we need to ensure the logging module is sending log data to the
log file. These changes ensure this happens correctly.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake/utils: Improve environment handling to allow UIs access to original environment

We need to empty out the environment whilst we build the cooker but
we need the environment for the UIs since hob uses DISPLAY and other
session variables.

This patch adapts the utils functions to return removed environment
components so we can reinject them for use by the UI, allowing hob
to work again.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ConfHandler: Improve regexp to fix mis-parsing of += and no whitespace

If you have:

FOO = "a"
FOO += "b"
FOO+= "c"

The expected result is "a b c" however we were seeing "a b" with the FOO+
variable being assigned the value "c". This isn't the expected result.

We need to make the name part of the variale non-greedy so that any + character
becomes part of the operator. This patch does that. I compared the configuration
in OE-Core before and after the change and only the test case changed.

[YOCTO #3834]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
Remove whitelisted vars from non-task deps

Though the value of variables in the BB_BASEHASH_WHITELIST is kept out of the
checksums, dependency on them is not, at least for variables and non-task
functions. In the code, the whitelist is removed from the overall task dep
list, but not the individual variable deps. The result of this is that
functions like sysroot_stage_all and oe_runmake end up with whitelisted
variables like TERM listed in their dependencies, which means that doing
a 'unset TERM' before building will result in all checksums for tasks that
depend on those changing, and shared state reuse not behaving correctly.

This is only really a potential issue for variables from the environment, as
it's the existance/removal of the variable that's an issue, not its value, and
the other whitelisted variables are set in our metadata. This which means in
practical terms the only cases where this is likely to be an issue are in
environments where one of the following are unset: TERM, LOGNAME, HOME, USER,
PWD, SHELL. This may seem like an unlikely circumstance, but is in fact a real
issue for those of us using autobuilders. Jenkins does not set TERM when
executing shell, which means shared state archives produced by your jenkins
server would not be fully reused by an actual user.

Fixed by removing the whitelisted elements from the individual variable deps,
not just the accumulated result.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
codeparser: Track appendVar and prependVar calls as we do for getVar

We need to track appendVar and prependVar calls just as we do for getVar in order
to ensure we're not missing variable dependencies.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
codeparser: Track bb.utils.contains usage

The bb.utils.contains function usage is getting increasingly used in the metadata
but isn't handled automatically by the python dependency tracking code. This patch
changes that and also adds the "OE" names for the functions.

Whilst there are reasons this is a bad idea, its likely outweighed by the
shear number of these references and the current holes in dependency information
which we're now relying heavily upon.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
wget: Only use -O option when not spidering

If we pass the -O option whilst spidering, empty files are created which is not
desired. We also need to ensure any subdirectories are created when using the
downloadfilename parameter.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
tests/codeparser: Hack around circular inclusion problem

There is a circular dependency issue where bb.data can't directly depend on
bb.parse. The tests were failing due to this issue which is misleading. This
patch hacks around it for now. I'd rather that than not running tests at all.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ConfHandler: Use re.X to make long regexp more readable

The __config_regexp__ in ConfHandler is quite long, and using re.X to
break the expression onto several lines make it a bit easier to read.

Signed-off-by: Olof Johansson <olof.johansson@axis.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Ensure expansions happen in URL parameters in uri_replace

For example, this allows us to specify downloadfilename=PATH in sstate
mirror urls to improve directory structures in DL_DIR when using sstate.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake & hob: reparse config files, when sanity check updates bblayers.conf

I've sent a patch to poky, with some changes in sanity.bbclass. When bblayers.conf
is updated, it updates also a variable to know when to reparse configuration files.
This patch contains the changes to reparse conf files, after a sanity check which
changes bblayers.conf.

[YOCTO #3213]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Improve lock/done stamp file paths for local files

Currently stamps end up at the top level of DL_DIR even if the files
themselves are in a directory structure. This patch preserves path
components allowing the top level directory to be less populated
which is an advantage for large sstate caches.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
monitordisk.py: disable the inode checking for some fs

There is an error when use disk monitor on btrfs:
WARNING: The free inode of rootfs is running low (0.000K left)
ERROR: Immediately abort since the disk space monitor action is "ABORT"!

This is beucase some fs formats' statvfs.f_files (inodes) is zero, thus
the statvfs.f_favail (free inodes) is zero, too, this a feature of the
fs, we disable the inode checking for such a fs.

[YOCTO #3609]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>