classes/package: fix FILES_INFO serialisation in pkgdata

The FILES_INFO entry in each pkgdata file stores the list of files for
each package. Make the following improvements to how this is stored:
* Store paths as they would be seen on the target rather than
  erroneously including the full path to PKGDEST (which is specific to
  the build host the package was built on)
* For simplicity when loading the data, store complete paths for each
  entry instead of trying to break off the first part and use it as the
  dict key
* Record sizes for each file (as needed by Toaster)
* Serialise the value explicitly using json rather than just passing it
  through str().

Fixes [YOCTO #5443].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
classes/package: record PKGSIZE as total file size in pkgdata

We were using "du -sk" to collect the total size of all files in each
package for writing out to PKGSIZE in each pkgdata file; however this
reports the total space used on disk not the total size of all files,
which means it is dependent on the block size and filesystem being used
for TMPDIR on the build host. Instead, take the total of the size
reported by lstat() for each packaged file, which we are already
collecting for FILES_INFO in any case.

Note: this changes PKGSIZE to be reported in bytes instead of kilobytes
since this is what lstat reports, but this is really what we should be
storing anyway so that we have the precision if we need it.

Fixes [YOCTO #5334]

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
classes/package: write PE and PKGE out to pkgdata

These are important parts of the version for every package, so we should
include them in PKGDATA just as we include PV/PR/PKGV/PKGR.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
classes/buildhistory: drop cruft from old SRCREV tracking implementation

This should have been removed when the implementation was rewritten in
OE-Core commit 2179db89436d719635f858c87d1e098696bead2a. The collected
values weren't being used anywhere since then.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
classes/buildhistory: add additional variables to image information

Add PACKAGE_EXCLUDE and NO_RECOMMENDATIONS to the info we track for
images, since these can change what ends up in the image.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
classes/buildhistory: improve collection of package info

Use a function added to SSTATEPOSTINSTFUNCS and read the necessary
information out of pkgdata, instead of using a function executed
during do_package that reads the data directly. This has two benefits:

* The package info collection will now work when the package content is
  restored from shared state

* Adding/removing the inherit of buildhistory will no longer change the
  do_package signatures and force re-execution of that function for
  every recipe.

Fixes [YOCTO #5358]

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
classes/buildhistory: reduce parsing time

Disable several python functions if not parsing within the worker
context. This avoids executing expensive operations while parsing
recipes (which is unnecessary).

(Thanks to Richard Purdie for pointing out the issue and suggesting the
workaround.)

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
classes/buildhistory: do git garbage collection after committing

We don't normally perform any operations (such as "git pull") that
trigger "git gc --auto", thus garbage collection never happens which
means performance of accessing the repository degrades noticeably over
time. Add an explicit "git gc --auto" to clean things up when needed.

Thanks to Elijah Newren and Ross Burton for suggesting this.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
scripts/oe-selftest: script to run builds as unittest against bitbake or various scripts

The purpose of oe-selftest is to run unittest modules added from meta/lib/oeqa/selftest,
which are tests against bitbake tools.

Right now the script it's useful for simple tests like:
  - "bitbake --someoption, change some metadata, bitbake X, check something" type scenarios (PR service, error output, etc)
  - or "bitbake-layers <...>" type scripts and yocto-bsp tools.

This commit also adds some helper modules that the tests will use and a base class.
Also, most of the tests will have a dependency on a meta-selftest layer
which contains specially modified recipes/bbappends/include files for the purpose of the tests.
The tests themselves will usually write to ".inc" files from the layer or in conf/selftest.inc
(which is added as an include in local.conf at the start and removed at the end)

It's a simple matter or sourcing the enviroment, adding the meta-selftest layer to bblayers.conf
and running: oe-selftest to get some results. It would finish faster if at least a core-image-minimal
was built before.

[ YOCTO #4740 ]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
meta-selftest: create a new test layer to be used by oe-selftest script

Everything in this layer is meant to be used by tests called by
scripts/oe-selftest. These are helper recipes/appends to test various bitbake
options or scripts.
Currently most of these files here only have "include test_recipe.inc" which
is the file tests will actually use.

Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/oeqa/selftest: buildoptions.py: add simple image build tests

Build images and tests different build options like RM_OLD_IMAGE
and for WARN_QA/ERROR_QA behaviour.

Signed-off-by: Alexandru Palalau <alexandrux.palalau@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/oeqa/selftest: add test modules for expected bitbake output and bitbake-layers

Tests for bitbake-layers and expected output for some bitbake options.

Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/oeqa: use the new manifest file for determining if a package is installed

Use the new manifest file instead of ${WORKDIR}/installed_pkgs.txt for determining
if an image has a certain package, because installed_pkgs.txt goes away with rm_work
enabled.
We can't use the IMAGE_MANIFEST var for the file path because that relies on IMAGE_NAME which
changes at every run (because of date), so we use the link which points to the last
one built.

[ YOCTO #5072 ]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python: do not replace ccache in the middle of a path

Python recipe did a sed s/ccache/$(CCACHE) on the Makefile, which
replaces all "ccache" including ones that consist of a full path.
This leads to build error when building in a project path with
"ccache" in its name. Fix it by only replacing "ccache " with
"$(CCACHE) ".

Signed-off-by: Lei Liu <lei.liu2@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
kmod: avoid parallel-tests

buildtest-TESTS and runtest-TESTS targets are required by ptest.
In order to have those targets in automake 1.13.4, serial-tests
should be specified since parallel test is assumed by default
and serial-tests is optional.

Signed-off-by: Tudor Florea <tudor.florea@enea.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
testimage: use the new targetcontrol.py module for running tests

This patch makes the necessary changes for using the targetcontrol.py module
so that one can run the same tests on a qemu instance or a remote machine
based on the value of TEST_TARGET variable: "qemu" or "simpleremote".
The default value is "qemu" which starts a qemu instance and it's the
with what we currently have.

With "simpleremote", the remote machine must be up with network and ssh
and you need to set TEST_TARGET_IP with the IP address of the remote machine
(it can still be a qemu instance that was manually started).

Basically testimage.bbclass now does something along the lines of:
 - load tests -> deploy (prepare) / start target -> run tests.
There were a couple of changes necessary for tests and
also some cleanups/renames that were needed to adjust this change. (use
ip everywhere when refering to target and server_ip when refering to host/build machine)
Also two unnecessary and unsed methods were dropped from sshcontrol.

[ YOCTO #5554 ]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/oeqa: targetcontrol.py: add abstraction for running tests on different targets

Add a new module which abstracts the target object used by testimage.bbclass

The purpose of this module is to move the deployment of a target from testimage.bbclass,
basically abstracting different implementations of how we setup a target and how it runs commands.
It allows to select one implementation or another by setting TEST_TARGET (currently to: "qemu" and "simpleremote").

QemuTarget is used to start a qemu instance (as it's currently done in testimage.bbclass)
SimpleRemoteTarget is meant for a remote machine (by setting TEST_TARGET_IP) that's already up and running
with network and ssh.
Simply put, it opens the door for running the tests on different types of targets by adding new classes
(maybe qemu-nfsroot or remote-special etc.). One could also override BaseTarget which currently uses
the existing SSHControl module and add a serial implementation.

[ YOCTO #5554 ]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
squashfs-tools: drop FILESPATHPKG

Drop FILESPATHPKG setting since it seems to be superfluous.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
squashfs-tools: set SUMMARY instead of DESCRIPTION

Also tidy up value a little bit.

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
alsa-utils-alsaconf: tidy up path setting

* Set FILESEXTRAPATHS instead of FILESPATH
* Don't set THISDIR, it's already set by base.bbclass

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Saul Wold <sgw@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>