bitbake: ensure -f causes dependent tasks to be re-run

If -f is specified, force dependent tasks to be re-run next time. This
works by changing the force behaviour so that instead of deleting the
task's stamp, we write a "taint" file into the stamps directory, which
will alter the taskhash randomly and thus trigger the task to re-run
next time we evaluate whether or not that should be done as well as
influencing the taskhashes of any dependent tasks so that they are
similarly re-triggered. As a bonus because we write this file as
<stamp file name>.taskname.taint, the existing code which deletes the
stamp files in OE's do_clean will already handle removing it.

This means you can now do the following:

bitbake somepackage
[ change the source code in the package's WORKDIR ]
bitbake -c compile -f somepackage
bitbake somepackage

and the result will be that all of the tasks that depend on do_compile
(do_install, do_package, etc.) will be re-run in the last step.

Note that to operate in the manner described above you need full hashing
enabled (i.e. BB_SIGNATURE_HANDLER must be set to a signature handler
that inherits from BasicHash). If this is not the case, -f will just
delete the stamp for the specified task as it did before.

This fix is required for [YOCTO #2615] and [YOCTO #2256].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: add -C option to invalidate a task and rebuild the target

This new command line option forces the specified task and all dependent
tasks up to the default task to re-run. This means that the following
single step:

bitbake -C compile somerecipe

is equivalent to the following two steps (with the recent change to -f):

bitbake -c compile -f somerecipe
bitbake somerecipe

Note that to work this option needs full hashing enabled (i.e.
BB_SIGNATURE_HANDLER must be set to a signature handler that inherits
from BasicHash). If this is not the case, -C effectively does nothing.

Based on a previous implementation of this option by Jason Wessel
<jason.wessel@windriver.com>.

Implements [YOCTO #2615].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/__init__: Make it clearer when uri_replace doesn't return a match

(From Poky rev: dc9976331c5cbb0983adb54f6deb97b9203bacbc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Fix error handling in uri_replace()

(From Poky rev: 1bfba28a583cb167f60e05ecdf34d0786dc1eec5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Only cache data if fn is set, its pointless caching it against a None value

(From Poky rev: c2df30bf6d1f8c263a38c45866936c1bf496ece5)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Ensure when downloading we are consistently in the same directory

This assists with build reproducuility. It also avoids errors if cwd
happens not to exist when we call into the fetcher. That situation
would be unusual but I hit it with the unit tests.

(From Poky rev: 86517af9e066c2da1d580fa66b7c7f0340f3403e)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Split try_mirrors into two parts

There are no functionality changes in this change

(From Poky rev: d222ebb7c75d74fde4fd04ea6feb27e10a862bae)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Explicitly check for mirror tarballs in mirror handling code

With support for things like git:// -> git:// urls, we need to be
more explicity about the mirrortarball check since we need to fall
through to the following code in other cases.

(From Poky rev: 28e858cd6f7509468ef3e527a86820b9e06044db)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Improve mirror looping to consider more cases

Currently we only consider one pass through the mirror list. This doesn't
catch cases where for example you might want to setup a mirror of a mirror
and allow multiple redirection. There is no reason we can't support this
and the patch loops through the list recursively now.

As a safeguard, it will stop if any duplicate urls are found, hence
avoiding circular dependency looping.

(From Poky rev: 0ec0a4412865e54495c07beea1ced8355da58073)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Simplify some looping constructs in uri_replace()

(From Poky rev: c6bd25150a842a530f958d7233b15ae50d42c6c8)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Remove basestring test and simplify uri_replace

(From Poky rev: d5657883d34bfef6beec594ac8d799f617b6b3ad)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Add parameter handling to uri_replace()

This means that parameters in the source expression are used as part of the
match. Parameters in the destination are used explicitly in the final
url.

(From Poky rev: c465cb0c5c927dd41d96ad6d6fa1566349574bb7)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: uri_replace() remove what amounts to a null operation and add some comments

(From Poky rev: 6d67200d052ba72258f5a0a178542ef99500a9cc)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: uri_replace() improve mirrortarball handling

We only consider mirror tarballs when the source and target urls are of
differing types. We also should clear all url paramters when handling
mirror tarballs.

(From Poky rev: da140b8b0b3dda5429f9eee68829ef5247cdfe12)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: uri_replace() only consider ud.localpath where its a file

Using ud.localpath as a basename when it points at a directory causes
problems. The supports_checksum() method gives a good indication of whether
ud.localpath can be used in the way we need.

(From Poky rev: 933ec8a44634e33f92f6f76de3a34094c3d63aa6)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
test/fetch: Add in unit tests for uri_replace() and git premirrors

(From Poky rev: 4be81b5cadb7813d37152411c23764501f5869d3)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
test/fetch: Add a couple of extra test cases

(From Poky rev: 01d8220ef8580ca980a14c05d500765ca2d37e2c)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Update replace_uri to handle uri types explicitly

For mirror mapping, we never use regexps for the type component of
the url. Doing so causes various slightly bizarre behaviour such as
https:// urls being translated to files:// urls which we have no handler
for. This patch forces the type matches to be direct, not regexp based
and gives the expected bahvriour.

(From Poky rev: 695d8aca0999d2d61970b990e83c3132ba6f12cb)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
ui/depexp.py: use the new progressbar in Hob

[Yocto 2149]

Use the new progressbar in Hob, so that we can reduce the maintenance
burden of the multiple GUIs.

Signed-off-by: Kang Kai <kai.kang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/git.py: Optimize clone fall back when it is local

A file:// url should use "clone -l" to greatly speed
up the clone in the case of a kernel when it is local.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>