bitbake: gitannex.py: Add Git Annex support

This add a Git Annex backend which reuses the Git fetcher code; it
allows managing files with git, without checking the file contents
into git, being useful when dealing with files larger than git can
currently easily handle, whether due to limitations in memory, time,
or disk space.

Signed-off-by: Otavio Salvador <otavio@ossystems.com.br>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: fetch2/git: add description for nobranch

Signed-off-by: Zhenhua Luo <zhenhua.luo@freescale.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
gitannex: Add missing file from previous commit

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/gitannex: Fix function arguments to match bitbake master

This code clearly uses an earlier fetcher API. Update it to match master.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
toaster: clone Simple UI as base for Toaster GUI

This patch clones the Simple UI to provide the base code for
the development of the Toaster GUI. The clone takes the place
of the application that was reserved for Javascript MVC code.

The templates used for Simple UI are renamed to start with
an "simple_" to prevent name resolution conflict with the
Toaster GUI templates.

Minor changes are made to the settings.py and urls.py in the
toaster main section to account for the newly enabled application.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toaster: update static media for Toaster GUI

In the process of making the GUI more usable, we
bring in the static assets used during the design
phase into the application.

Licenses used:

    Bootstrap 2.3.2 under Apache License 2.0
    Font Awesome under SIL OFL 1.1 (GPL compatible) and MIT licence
    GLYPHICONS Halflings released with Bootstrap, CC BY 3.0
    jQuery 2.0.3 under MIT licence
    prettify.js under Apache Licence 2.0

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toaster: Create the base page navigation structure

Updating the general container pages to use the graphical
design and features from the design phase.

In the process of adapting the Simple UI to the designed
interface, we create all the pages and the navigation
structure for the Toaster GUI.

Views for each page have been added, and the url mapping
has been updated to reflect newly added pages.

The table page has been refactored to be component-oriented
instead of class-oriented in order to facilitate reusage.

Changes are made in different layers of the template
(base, basetable) in order to maximize code reuse among
different pages in the build.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toaster: Build dashboard implementation

This patch adds the build dashboard page implementation,
which is the landing page for the Toaster GUI.

Also adds correct links from the main build page
to the various parts of the dashboard.

    [YOCTO #4258]

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
toaster: Toaster GUI, generic search, filter and order

This patch implements table searching, filtering and ordering, in a
generic mode reusable for all tables.

The search operates list of fields defined in the corresponding
class for each model, search_allowed_fields.

The search expression and filters are sent through GET requests
using a QuerySet-like input. The inputs are filtered and
validated before usage to prevent inadvertent or malicious use.

Filters and table headers are defined in the views for each table,
and rendered by generic code which is easily modified for various
tables.

The Build table and Configuration table are implemented using this
framework as an example of how it should be used.

    [YOCTO #4249]
    [YOCTO #4254]
    [YOCTO #4255]
    [YOCTO #4256]
    [YOCTO #4257]
    [YOCTO #4259]
    [YOCTO #4260]

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
siggen: Remove fsync that is causing performance issues

This fsync was added for belt and braces protection for things like sstate
on NFS. To be honest, it probably doesn't buy much, if the rename isn't
atomic, all bets are off anyway and there are bigger issues to worry about.

The issue is that at the end of every task, the dump_sig() code is triggered
to save out information about the task and this was triggering an fsync(fd).

Whilst it may select the file descriptor, on file systems like ext4, it will
require large parts of the journal to be written out so it can have
significant impact. latencytop showed an average fsync() call overhead of about
2s and if that happens for 5000 tasks, the time mounts up. This blocks the next
task execution by that time.

We therefore drop the fsync since in reality its causing problems and is unlikely
to buy much.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Don't allow '/' in user:pass, fix branch containing '@'

* currently decode_url regexp parses branch=@foo as username so it ends like this:
  - ('git', '', 'foo', 'git.openembedded.org/bitbake;branch=', '', {})
  + ('git', 'git.openembedded.org', '/bitbake', '', '', {'branch': '@foo'})
* http://hg.python.org/cpython/file/2.7/Lib/urlparse.py also assumes
  that there is at least one '/' as separator between netloc and path,
  params, so it looks reasonable to prevent including '/' in username

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/gitsm: Use ud.basecmd instead of hardcoding git

This allows FETCHCMD_git to override the fetcher command as the git fetcher does.

[YOCTO #5717]

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
SignatureGenerator: Add empty implementation for dump_sigs

* bitbake -S throws exception when 'noop' signature handler is used

[YOCTO #5738]

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/__init__.py: let try_mirror_url return correct value

The fetcher will try:

1) PREMIRROR
2) Upstream
3) MIRROR

If it fails to download from the Upstream, but succeeds from the MIRROR,
and ud.localpath != origud.localpath (for example, the git tarball),
then we will get the error (e.g.: xf86-video-omapfb):

ERROR: Function failed: Fetcher failure for URL: 'xxx'. Unable to fetch URL from any source.
ERROR: Logfile of failure stored in: /path/to/log.do_fetch.28024

It should not show the error and let the build go on since it succeeds.
(e.g.: xf86-video-omapfb)

[YOCTO #5686]

Signed-off-by: Robert Yang <liezhi.yang@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Clean up srcrev_internal_helper

Currently INVALID and None are checked as incorrect values under different
circumstances. This code standardises those checks to be consistent. We
should phase out the use of "INVALID".

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Sanity check SRCREV matches rev/tag parameter

Add a sanity check so that if some SRCREV is set and a rev parameter is given
to the url, the revision given should match.

Any tag parameter behaves the same as rev. If both are specified, error to
tell the user we're confused rather than do something which may or may not
be what they intended.

Also add some unittests for this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Improve invalid SRCREV error message

The current message can be ambiguous, improve it (and also rename a
variable to clean up the rest of the function).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/git: Anchor names when using ls-remote

When specifying tags, they're searched for unanchored so foo/bar could
match:

refs/heads/abc/foo/bar
refs/heads/xyz/foo/bar
refs/heads/foo/bar

This change anchors the expressions so they are based against heads
or tags (or any other base level tree that has been created).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2/git: Dereference unresolved tags with ls-remote

We need to deference tags when trying to map them to commit IDs with
ls-remote. If we don't do this, a given commit might not show up
later in a specific branch. There appears to be no good reason not
to do this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>