sstate/path.py: Add copyhardlinktree() function and use for performance optimisation

Add a function which copys a tree as a set of hardlinks to the original
files, then use this in sstate to reduce some of the overhead of sstate
package creation since the file isn't actually copied.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
sstate.bbclass: Ensure build directory is cleaned to start with

This directory is cleaned upon completion however if a previous build
crashes, it can lead to corrpution, hence ensure its clean at the start
too.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: Multithread per file dependency generation code

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package: Don't export PATH

PATH is already exported, we don't need to do this each time we run
something, its just noise and overhead.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package: Process package stripping in parallel

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: use the multiprocess pool from bitbake

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
insane.bbclass: Add pkgvarcheck to check for suboptimal usages of variables

Check through the variables:
'RDEPENDS', 'RRECOMMENDS', 'FILES', 'pkg_preinst', 'pkg_postinst', 'pkg_prerm', 'pkg_postrm'
and if there is a variable set which isn't package specific, inform the user
of this.

Using these variables without a package suffix is bad practise and complicates
dependencies of packages unnecessarily as well as complicates the code. Lets
convert the remaining issues and then we can take the small performance gain.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
insane.bbclass: Add documentation headers for logical code blocks

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
staging/insane.bbclass: Move legacy do_stage check iinto insane.bbclass

We might as well put all the sanity checks in one place.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
staging.bbclass: Drop unused/legacy function

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
update-rc.d: Drop OVERRIDES code

The data store copy and overrides is overkill given the small number
of accesses that are being made. This simplifies the code.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: Fix up bb.mkdirhier/bb.copyfile usage

These are in bb.utils so lets the correct function and avoid the overhead
of the fixup/warning for the deprecated usage.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: Rewrite split_and_strip_files

The split_and_strip_files funciton was hard to follow and its usage of prefixes
to strings was unusual. This rewrites it to use a list of hardlinks, symlinks and
elffiles where each list is iterated over at the correct point.

This means we can avoid creating dandling symlinks for example so we can simply
delete the cleanup code for this.

The isfile() check is also removed which gives a significant improvement in speed.
Its uneeded since os.walk will have already checked things in files are files.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
kernel.bbclass: Improve populate_packages_prepend

Small performance tweaks for populate_packages_prepend:

* Compile the regexps once at the start
* Don't keep importing a module which is already imported
* No need to check PKG is set, we'd have failed long before now if it wasn't
* Don't export PATH, bitbake takes care of this at the task level

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: Make use of cleandirs and dirs function flags

We can use the cleandirs and dirs flags for the fuctions to handle
directory cleaning and creation at the bitbake level rather than
using these calls within the functions

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: Various minor performance tweaks

* Call getVar outside the loop
* Drop unneeded PATH export (bitbake does this already)
* Drop unused variable
* Simplify if statement nesting
* Simplify variable expandion to a getVar call (expand would just call getVar)

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: Simplify empty directory removal

Rather than an exec() per directory, we might as well exec one command and
be done with it.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: Add PACKAGESPLITFUNCS variable

Prepending to populate_packages is rather ugly and means its hard to trace
errors and also profiling informaiton is summed together in one function.

This patch starts to split out the prepends to become separate functions
to avoid these issues. This is generally a neater way to write functions
than prepending to where there can sometimes be variable scope issues
and we've been bitten by whitespace issues in the past.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: Better document the different phases of operation

Add headers to document the different phases of do_package and
make the steps clearer.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
package.bbclass: Pre-expand some variables to save time

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>