runqueue: add runQueueTaskSkipped event

Adding a runQueueTaskSkipped to notify that the tasks that are not
run either because they are set-scened or they don't need an update
(timestamp was ok).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker: Avoid duplication for taskdata creation

Clean-up to avoid duplication and promote code reuse to factor
taskdata creation into a common function.

[RP: minor tweaks]

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker: Allow profiling of the parser in profile mode

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data: Be explicit in data_db check

The if statement current causes the size of parent to be calcuated which
is like a len() operation on a datastore. Since we're only interested
whether the value is none, checking explictly for this gives a
small performance gain.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
siggen: Use lookup cache exclusively

All the values we need are already guaranteed to be in the lookupcache
so rather than fetch variables again, just use the cache. This gives a
small performance improvement and simplifies the code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: Improve variable expansion regexp

Debugging showed the variable expansion regexp was catching python
expressions (starting with @). Since these are caught by their own
dedicated regexp, stop matching these for the plain variable expansion
for small performance improvements.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: use the expand_cache in VariableParse

When in VariableParse, use the expand_cache if possible rather than looking
up data. Ultimately it would come from the same place but this short cuts
a heavily used code block for speed improvements.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data: Use direct iteration, not keys()

Profiling shows the creation of keys() has overhead and we're better using
an iterator rather than the memory associated with the huge list of keys
when iterating the whoe datastore. We minimise the number of times
we do this to twice only per recipe.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data: Cache an list of export variables

Compute a cache of the list of potential export variables so
that we don't have to compute the list from scratch.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: Add explict None checks

Simple if xxx checks end up calling len(xxx). We're interested in the specific case
of None which means we can break out the iterator much earlier after the first
item. This adds in the specific tests for None in what is a hot path in the
data store code which gives small performance gains.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: Allow expansion of flags in getVarFlags

Allow a list of flags to expand to be passed into getVarFlags. This
is useful within bitbake itself to optimise performance of the
dependency generation code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: Allow flags to use the expand cache

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: Cache the fact a variable accesses another even if its unset

If a variable references another but it isn't set at present, the
reference wasn't stored. It really should be marked as a reference
and the higher level dependency code can handle as appropriate.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data: Optimise build_dependencies a little

Instead of multiple calls to getVarFlag, make one call to getVarFlags, only expanding
the flags that need to be expanded. This improves performance.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data: Optimise flag lookup in build_dependencies

When looking up flag variable dependencies, large chunks of the function
aren't needed. Optimise the function flow accordingly for speed.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data: Optimise flag exclusion list handling

Move the variable lookup to the outer loop for performance, replacing
a now unneeded parameter (after the previous changes).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bb.fatal: Raise a BBHandledException instead of exiting

With new bitbake UIs having the cooker exit at 'random' points
in the codebase is problematic. This patch raises an exception
which matches the siutation instead.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>