scripts/oe-selftest: script to run builds as unittest against bitbake or various scripts

The purpose of oe-selftest is to run unittest modules added from meta/lib/oeqa/selftest,
which are tests against bitbake tools.

Right now the script it's useful for simple tests like:
  - "bitbake --someoption, change some metadata, bitbake X, check something" type scenarios (PR service, error output, etc)
  - or "bitbake-layers <...>" type scripts and yocto-bsp tools.

This commit also adds some helper modules that the tests will use and a base class.
Also, most of the tests will have a dependency on a meta-selftest layer
which contains specially modified recipes/bbappends/include files for the purpose of the tests.
The tests themselves will usually write to ".inc" files from the layer or in conf/selftest.inc
(which is added as an include in local.conf at the start and removed at the end)

It's a simple matter or sourcing the enviroment, adding the meta-selftest layer to bblayers.conf
and running: oe-selftest to get some results. It would finish faster if at least a core-image-minimal
was built before.

[ YOCTO #4740 ]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
meta-selftest: create a new test layer to be used by oe-selftest script

Everything in this layer is meant to be used by tests called by
scripts/oe-selftest. These are helper recipes/appends to test various bitbake
options or scripts.
Currently most of these files here only have "include test_recipe.inc" which
is the file tests will actually use.

Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/oeqa/selftest: buildoptions.py: add simple image build tests

Build images and tests different build options like RM_OLD_IMAGE
and for WARN_QA/ERROR_QA behaviour.

Signed-off-by: Alexandru Palalau <alexandrux.palalau@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/oeqa/selftest: add test modules for expected bitbake output and bitbake-layers

Tests for bitbake-layers and expected output for some bitbake options.

Signed-off-by: Corneliu Stoicescu <corneliux.stoicescu@intel.com>
Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/oeqa: use the new manifest file for determining if a package is installed

Use the new manifest file instead of ${WORKDIR}/installed_pkgs.txt for determining
if an image has a certain package, because installed_pkgs.txt goes away with rm_work
enabled.
We can't use the IMAGE_MANIFEST var for the file path because that relies on IMAGE_NAME which
changes at every run (because of date), so we use the link which points to the last
one built.

[ YOCTO #5072 ]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
python: do not replace ccache in the middle of a path

Python recipe did a sed s/ccache/$(CCACHE) on the Makefile, which
replaces all "ccache" including ones that consist of a full path.
This leads to build error when building in a project path with
"ccache" in its name. Fix it by only replacing "ccache " with
"$(CCACHE) ".

Signed-off-by: Lei Liu <lei.liu2@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
kmod: avoid parallel-tests

buildtest-TESTS and runtest-TESTS targets are required by ptest.
In order to have those targets in automake 1.13.4, serial-tests
should be specified since parallel test is assumed by default
and serial-tests is optional.

Signed-off-by: Tudor Florea <tudor.florea@enea.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
testimage: use the new targetcontrol.py module for running tests

This patch makes the necessary changes for using the targetcontrol.py module
so that one can run the same tests on a qemu instance or a remote machine
based on the value of TEST_TARGET variable: "qemu" or "simpleremote".
The default value is "qemu" which starts a qemu instance and it's the
with what we currently have.

With "simpleremote", the remote machine must be up with network and ssh
and you need to set TEST_TARGET_IP with the IP address of the remote machine
(it can still be a qemu instance that was manually started).

Basically testimage.bbclass now does something along the lines of:
 - load tests -> deploy (prepare) / start target -> run tests.
There were a couple of changes necessary for tests and
also some cleanups/renames that were needed to adjust this change. (use
ip everywhere when refering to target and server_ip when refering to host/build machine)
Also two unnecessary and unsed methods were dropped from sshcontrol.

[ YOCTO #5554 ]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/oeqa: targetcontrol.py: add abstraction for running tests on different targets

Add a new module which abstracts the target object used by testimage.bbclass

The purpose of this module is to move the deployment of a target from testimage.bbclass,
basically abstracting different implementations of how we setup a target and how it runs commands.
It allows to select one implementation or another by setting TEST_TARGET (currently to: "qemu" and "simpleremote").

QemuTarget is used to start a qemu instance (as it's currently done in testimage.bbclass)
SimpleRemoteTarget is meant for a remote machine (by setting TEST_TARGET_IP) that's already up and running
with network and ssh.
Simply put, it opens the door for running the tests on different types of targets by adding new classes
(maybe qemu-nfsroot or remote-special etc.). One could also override BaseTarget which currently uses
the existing SSHControl module and add a serial implementation.

[ YOCTO #5554 ]

Signed-off-by: Stefan Stanacar <stefanx.stanacar@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>