bitbake server: create common server infrastructure

In an attempt to minimize code duplication, create
clear interfaces, and maximize code reuse through OOP,
bb.server adds base classes for the BitBakeServer,
BitBakeServerConnection and actual server implementations
instructed in particular server types.

These classes document the minimum interfaces that the
derived classes must implement, and provide boilerplate code.

Changes to None, Process and XMLRPC servers as to use
the common server infrastructure.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
xmlrpc: Allow server to remain memory resident

This patch enables the XMLRPC server to remain
resident in memory after a task is run, and to accept
a new controlling client. To check the server after
task completion, do

lsof bitbake.lock

in the build directory. Kill the server with kill.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: Improve remote server handling

Two fixes in bitbake related to running remote servers -
* can now specify correctly the bind port
* the information in print conforms to common server infrastructure

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
.gitignore: add vim tag and swp files

Modify gitignore as to include popular editor's temporary
files, and lookup tags file.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cooker: Split data from configuration

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
lib/bb/cache.py: invalidate cache when file checksum entry no longer exists

Go through the cached list of file checksums and check if any of the
files no longer exist; if any are missing then invalidate the cached
recipe, which will force it to be reparsed and thus force the list of
files to be collected again. This prevents a warning when moving a file
to a different location that is still picked up by the recipe, e.g.
moving a file from a "files" subdirectory to one named with the recipe
name (${BPN}).

Fixes [YOCTO #4474].

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
tinfoil: fix for move of data attribute to cooker

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-layers: fix for move of data object to cooker

Signed-off-by: Paul Eggleton <paul.eggleton@linux.intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
fetch2: Fix AUTOINC handling

AUTOINC was meant to appear once at the start of the version string.
The list of names may not be sorted meaning it could get inserted in
the middle. This patch simplifies the code and ensures it appears at
the start.

Include cache version bump to ensure the cache picks up these changes.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
runqueue: Split runqueue to use bitbake-worker

This is a pretty fundamental change to the way bitbake operates. It
splits out the task execution part of runqueue into a completely
separately exec'd process called bitbake-worker.

This means that the separate process has to build its own datastore and
that configuration needs to be passed from the cooker over to the
bitbake worker process.

Known issues:

* Hob is broken with this patch since it writes to the configuration
  and that configuration isn't preserved in bitbake-worker.
* We create a worker for setscene, then a new worker for the main task
  execution. This is wasteful but shouldn't be hard to fix.
* We probably send too much data over to bitbake-worker, need to
  see if we can streamline it.

These are issues which will be followed up in subsequent patches.

This patch sets the groundwork for the removal of the double bitbake
execution for psuedo which will be in a follow on patch.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
runqueue: Move the bitbake-worker execution to a higher level

The worker was being executed by each execution queue so would get
constructed twice for each build. This is wasteful so move execution
to the main runqueue so we only have to start the worker once.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
runqueue: Abstract the start and teardown worker functions

We're going to need a fakeroot/pseudo version of the worker so
abstract the code to start the worker process.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
runqueue: Spawn a separate worker for fakeroot tasks

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv: Adapt autostart to bitbake-worker

With the change to bitbake-worker we need to ensure the workers know
how to contact the PR service, the magic 0 port and singleton is
no longer enough.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: Bump minimum python version to 2.7.3

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: Update version to 1.19.1

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: Add event mask flag support

Add a flag to event handlers which lists the events a given handler wishes to
process. By default event handlers recieve all events but this means
we can stop running code in many cases if we know it doesn't want the event.

This is part of the fix for YOCTO #3812, but implements filtering only
for class event handlers; the other part (events filter for UIs) will be
the subject of a different patch.

Signed-off-by: Bogdan Marinescu <bogdan.a.marinescu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
xmlrpc: removes the event servers controller

XMLRPC API ran a XMLRPC server at port+2 in order
to provide endpoints to register an event server
in which to dump the events.

This is no longer used, so we remove it.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
runqueue: Improve handling of failing setscene tasks with hard dependencies

If a setscene task has a hard dependency on a task like pseudo-native, its
expected that the setscene task will not run unless the dependency is met.

This adds code to ensure that is the case, otherwise a bug would show up
with a usecase like:

bitbake gnome-common
bitbake pseudo-native -c cleansstate
bitbake gnome-common -c clean
bitbake gnome-common

With the double wrapper script environment, we'd not see issues like
this as it would be masked. The problem theoretically affects code like
useradd too as well as anything using a sstate postinstall.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
compat/server/utils: Jettison pre python 2.7.3 workarounds

Now we've moved to require python 2.7.3, we can jettison the compatibility
workarounds/hacks for older python versions.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>