siggen: Remove fsync that is causing performance issues

This fsync was added for belt and braces protection for things like sstate
on NFS. To be honest, it probably doesn't buy much, if the rename isn't
atomic, all bets are off anyway and there are bigger issues to worry about.

The issue is that at the end of every task, the dump_sig() code is triggered
to save out information about the task and this was triggering an fsync(fd).

Whilst it may select the file descriptor, on file systems like ext4, it will
require large parts of the journal to be written out so it can have
significant impact. latencytop showed an average fsync() call overhead of about
2s and if that happens for 5000 tasks, the time mounts up. This blocks the next
task execution by that time.

We therefore drop the fsync since in reality its causing problems and is unlikely
to buy much.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Don't allow '/' in user:pass, fix branch containing '@'

* currently decode_url regexp parses branch=@foo as username so it ends like this:
  - ('git', '', 'foo', 'git.openembedded.org/bitbake;branch=', '', {})
  + ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'})
* http://hg.python.org/cpython/file/2.7/Lib/urlparse.py also assumes
  that there is at least one '/' as separator between netloc and path,
  params, so it looks reasonable to prevent including '/' in username

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/gitsm: Use ud.basecmd instead of hardcoding git

This allows FETCHCMD_git to override the fetcher command as the git fetcher does.

[YOCTO #5717]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SignatureGenerator: Add empty implementation for dump_sigs

* bitbake -S throws exception when 'noop' signature handler is used

[YOCTO #5738]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/__init__.py: let try_mirror_url return correct value

The fetcher will try:

1) PREMIRROR
2) Upstream
3) MIRROR

If it fails to download from the Upstream, but succeeds from the MIRROR,
and ud.localpath != origud.localpath (for example, the git tarball),
then we will get the error (e.g.: xf86-video-omapfb):

ERROR: Function failed: Fetcher failure for URL: 'xxx'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /path/to/log.do_fetch.28024

It should not show the error and let the build go on since it succeeds.
(e.g.: xf86-video-omapfb)

[YOCTO #5686]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Clean up srcrev_internal_helper

Currently INVALID and None are checked as incorrect values under different
circumstances. This code standardises those checks to be consistent. We
should phase out the use of "INVALID".

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Sanity check SRCREV matches rev/tag parameter

Add a sanity check so that if some SRCREV is set and a rev parameter is given
to the url, the revision given should match.

Any tag parameter behaves the same as rev. If both are specified, error to
tell the user we're confused rather than do something which may or may not
be what they intended.

Also add some unittests for this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Improve invalid SRCREV error message

The current message can be ambiguous, improve it (and also rename a
variable to clean up the rest of the function).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/git: Anchor names when using ls-remote

When specifying tags, they're searched for unanchored so foo/bar could
match:

refs/heads/abc/foo/bar
refs/heads/xyz/foo/bar
refs/heads/foo/bar

This change anchors the expressions so they are based against heads
or tags (or any other base level tree that has been created).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/git: Dereference unresolved tags with ls-remote

We need to deference tags when trying to map them to commit IDs with
ls-remote. If we don't do this, a given commit might not show up
later in a specific branch. There appears to be no good reason not
to do this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/wget: Check downloaded file isn't zero size

I can't think of a reason we'd download zero sized files however there are
reasons zero length files can accidently make it onto source mirrors.

This check allows us to ignore the broken files and switch to another
mirror rather than fail with odd checksum failures.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
runqueue: Only attempt to print closest matching task if there is a match

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>