process: Further improve robustness against server shutdown

Currently, if an exception occurs in an event handler, the server shuts
down but the UI simply hangs. This happens in two places, firstly waiting
for events and secondly, sending events to a server which no longer exists.

The latter does time out, the former does not. These patches improve
both code sections to check if the main server process is alive and if not,
trigger things to shut down gracefully. This avoids the timeout in the
command sending case too.

This resolves various cases where the UI would simply hang indefintely.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
knotty: Improve exception handling

Currently, IOErrors are just passed over due to the broken Exception
clause. A command like "bitbake X | <invalid command>" would break stdout
triggering a traceback. With these changes we print the exceptions, shut down
the server gracefully and exit which is a much nicer behaviour and is less
confusion to the user.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
command: Trigger updateCache to shut down any active parser threads

If we trigger a shutdown whilst parsing for whatever reason, in some
cases we were not closing down the parser threads. This change
ensures we do so. The function names are not entirely intuative
but the behaviour is more correct (and commented). The previous bug
with the stdout failure would trigger this one, if there was a cold
cache and parsing was required (but not otherwise).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-worker: Improve sigterm handler

When processes terminate, we really want all of the child processes to
terminate too. This was not happening for worker processes which spawned their
own multiprocessing pools, leading to build hangs. This change ensures any
sigterm gets passed to the whole process group. In local tests, this resolved
some hanging process workloads I could generate. It does rely on signals
being delivered in a timely fashion and there is a multiprocessing bug we have
to work around there.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
utils: Add workaround for multiprocessing bug

Our usage of multitprocessing is problematic. In particular, there is a bug
in python 2.7 multiprocessing where signals are not handled until command
completion instead of immediately.

This adds a workaround into our wrapper function to deal with the issue.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
world-image: add extra target

* similar to world target, but collects all PACKAGES
* recipes marked with EXCLUDE_FROM_WORLD are excluded from world as well
  as world-image
* recipes marked with EXCLUDE_FROM_WORLD_IMAGE are excluded completely
  from world-image, sometimes you can build 2 recipes in same sysroot,
  but cannot install them in the same image
* PACKAGES listed in EXCLUDE_PACKAGES_FROM_WORLD_IMAGE are also
  excluded. You can remove e.g. ${PN}-dbg, but keep all other PACKAGES.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
process: Deal with infinite looping of the server

Currently if an exception occurs, we just run the idle handler again and
again, usually looping indefinitely. Chances are the exception that occurred
will keep occurring and this is not a good place to be.

This was breaking the autobuilders with gigabytes of logs.

At least improve things so the cooker shuts down gracefully when this happens.
Some trace of the original problem may still be present on the console too!

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>