process: Deal with infinite looping of the server

Currently if an exception occurs, we just run the idle handler again and
again, usually looping indefintely. Chances are the exception that occurred
will keep occuring and this is not a good place to be.

This was breaking the autobuilders with gigabytes of logs.

At least improve things so the cooker shuts down gracefully when this happens.
Some trace of the original problem may still be present on the console too!

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
knotty: Improve exception handling

Currently, IOErrors are just passed over due to the broken Exception
clause. A command like "bitbake X | <invalid command>" would break stdout
triggering a traceback. With these changes we print the exceptions, shut down
the server gracefully and exit which is a much nicer behaviour and is less
confusion to the user.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake-worker: Improve sigterm handler

When processes terminate, we really want all of the child processes to
terminate too. This was not happening for worker processes which spawned their
own multiprocessing pools, leading to build hangs. This change ensures any
sigterm gets passed to the whole process group. In local tests, this resolved
some hanging process workloads I could generate. It does rely on signals
being delivered in a timely fashion and there is a multiprocessing bug we have
to work around there.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
command: Trigger updateCache to shut down any active parser threads

If we trigger a shutdown whilst parsing for whatever reason, in some
cases we were not closing down the parser threads. This change
ensures we do so. The function names are not entirely intuitive
but the behaviour is more correct (and commented). The previous bug
with the stdout failure would trigger this one, if there was a cold
cache and parsing was required (but not otherwise).

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
utils: Add workaround for multiprocessing bug

Our usage of multitprocessing is problematic. In particular, there is a bug
in python 2.7 multiprocessing where signals are not handled until command
completion instead of immediately.

This adds a workaround into our wrapper function to deal with the issue.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
process: Further improve robustness against server shutdown

Currently, if an exception occurs in an event handler, the server shuts
down but the UI simply hangs. This happens in two places, firstly waiting
for events and secondly, sending events to a server which no longer exists.

The latter does time out, the former does not. These patches improve
both code sections to check if the main server process is alive and if not,
trigger things to shut down gracefully. This avoids the timeout in the
command sending case too.

This resolves various cases where the UI would simply hang indefintely.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
siggen: Fix a subtle bug in hash calculation for shared work tasks

With a shared work task like gcc, the task can be run from a variety of
different recipes which may have different virtual extensions in place.

Depending on whether gcc-runtime or nativesdk-gcc-runtime's do_preconfigure
task is called for example will change the sorting of the task hashes due
to the way clean_basename currently works.

The correct thing to do here is sort on the base filename first, then any
extension when ordering the hashes. This means we do account for things
like recipes with both a native and non-native dependency but we also fix
the shared work case where we don't care whether it was a virtual version
or not.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>