build.py: Provide useful diagnostics when exiting.

Running scripts with 'set -e' produces silent failures with no
diagnostic. Add an exit handler which produces diagnostics, including
details of what was running if the shell seems to be bash.

Signed-off-by: Peter Seebach <peter.seebach@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
propertydialog.py : Scrollable windows for long tooltips

In some cases, the length of the description and the brought
in by field was too big. That led to the size of the property
dialog exceeding Hob's size. For long tooltips we use
scrollable windows now.

[HOB #4321]

Signed-off-by: Andrei Dinu <andrei.adrianx.dinu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob: changes to image combo box from image configuration page

In some cases when a new machine is selected, the image combo
shows the same option. Fixed this issue, because the image
combo box should be reseted.

Fix the counter for the options in the combobox. This bug
was introduced by the templates functionality. The combo box had some
last changes, and I forgot about this counter.

[YOCTO #4858 & #5000]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
build.py: create symlink for run.do_xxx scripts

The 'courtesy' symlink for log.do_xxx are quite useful when debugging, so
with this commit, we now get similar 'courtesy' symlink for run.do_xxx
scripts.

We only create symlink for tasks, not individual functions.

The symlink is create right before the actual runfile is created, indeed
we cannot create the symlink right after running the task since a failure
or execption can happen, in which case the symlink wouldn't be created,
and symlink are particularely useful when the task failed!

Another option would be create the symlink after the runfile is created,
and before the script is executed, but that means we need to duplicate the
code in case of Shell vs Python task.

Signed-off-by: Nicolas Dechesne <nicolas.dechesne@linaro.org>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
event/msg: Add primitive server side UI log record filtering

Currently one of the bigger bottlenecks in bitbake is passing all the
log messages over IPC to the UI. This is worthwhile if the UI is going
to use them, pointless otherwise. The memory resident bitbake suffers
from this performance issue particularly badly.

This patch filters the log events on the server side with the global
log levels and hence reduces the traffic. This speeds up parsing
(18.5s down to 17s) and bitbake general command overhead is reduced
(7.3s for a NOP to 6.2s).

What isn't added here is general event filtering or the ability to
change the log levels once set. Provision is made for adding this
in a follow up patch though.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv/serv: Fix pid file removal

Mark Hatle spotted there were pid files being left around. This patch
fixes things so the removal function is called correctly, the code
contained a typo.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
parse: Return IOError when including file with absolute path

* resolve_file was behaving different when relative and absolute
  paths were passed to it

* include relative-path/non-existent-file.inc
  works correctly resolve_file throws IOError, BBHandler.py:handle()
  doesn't catch it, ConfHandler.py:include() catches IOError and shows:
  DEBUG: CONF file 'relative-path/non-existent-file.inc' not found
* include /absolute-path/non-existent-file.inc
  was failing, because resolve_file just returns fn,
  BBHandler.py:handle() calls bb.parse.mark_dependency(d, abs_fn)
  which throws:
  OSError: [Errno 2] No such file or directory: '/absolute-path/non-existent-file.inc'
  and parsing fails.
  Ad isfile() test for absolute fn and throw IOError to make
  resolve_file behavior consistent for both paths.

* I know we had some issues with -b relative-path-to-recipe.bb and
  absolute path, so consider this patch only as RFC and documentation of
  this problem

* Catch OSError too in ConfHandler.py:include() e.g. in case the file exists, but user
  cannot read it or something like that.

Signed-off-by: Martin Jansa <Martin.Jansa@gmail.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
process: Improve exit handling and hangs

It turns out we have a number of different ways the process server termination can
hang. If we call cancel_join_thread() on the event queue, it means that it can be left
containing partial data. This means the reading of the event queue in the terminate()
function can hang, the timeout and block parameters to Queue.get() don't make any
difference.

Equally, if we don't call cancel_join_thread(), the join_thread in terminate()
will hang giving a different deadlock.

The best solution I could find is to loop over the process is_alive() after requesting
it stops,  trying to join the thread and if that fails, try and flush the event
queue again.

It wasn't clear what difference a force option should make in this case, we're
gracefully trying to empty queues and shut down regardless of whether its a SIGTERM
so I've simply removed the force option.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
server/xmlrpc/prserv: Add sane timeout to default xmlrpc server

The standard python socket connect has long timouts which make sense for remote
connections but not local things like the PR Service. This adds a timeout
parameter to the common xmlrpc server creation function and sets it to a more
reasonable 5 seconds.

Making the PR server instantly exit is a good way to test the effect of this
on bitbake.

We can remove the bodged timeout in the PRServer terminate function which
has the side effect of affecting global scope.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake: Add ui event handlers filtering

Add functionality to allow UIs to update and change the types of events they
recieve. To do this we need to add a new command and also need to be able
to obtain the current event hander ID. In the case of xmlrpc, this is
straightforward, in the case of the process server we need to save the result
in a multiprocessing.Value() so we can retrive it. An excplit command
was added to the server API to facilitate this.

The same function can also be used to mask or unmask specific log messages,
allowing the UI to optionally differ from the standard set of message
filtering.

Based upon work by Cristiana Voicu <cristiana.voicu@intel.com>

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: Add _remove operator

There are long standing complaints about the fact its very difficult
to remove a portion of a variable. The immediate request is for a -=
and =- operator. The trouble is that += and =+ are "immediate"
operators and are applied straight away. Most people would expect
-= and =- to be deferred to have the effect most people desire and
therefore implementing -= and =- would just make the situation more
confusing.

This deferred operation is much more similar to the override syntax
which happens at data store finalisation. The _remove operator is
therefore in keeping with the _append and _prepend operations.

This code is loosely based on a patch from Peter Seebach although it
has been rewritten to be simpler, more efficient and avoid some
potential bugs.

The code currently only works on space delimited variables, which
are by far the most commom type. If bitbake is ehanced to support
types natively in future, we can adjust this code to adapt to that.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
command.py: Call updateCache for all states != running

updateCache handles the logic for shutting down the parsing so we need
to call it for all cases when we're not running.

This fixes hangs if Ctrl+C is pressed during parsing.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
bitbake/event.py: UIhandler filter should work without a mask

The default for the mask will be * (all the handlers)

Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob: add event handlers filtering in Hob

Create the _evt_list for hob; it is longer than the knotty
uses because it handles more events.

Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob: fixes for image combo box

When an image from scratch is selected, and recipes parsing
is canceled, the image shown by the combo box isn't correct.

[YOCTO #5000]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
serv.py: Fix hang when spawned dynamically with bitbake

The PRServer has the possibility to hang indefinitely blocking on a
semaphore processing a xmlrpc request to send an event back to the
main bitbake instance.  This was observed during a "bitbake -e" on a
heavily loaded machine and the main bitbake instance and cooker exited
before the PRServer emitted its first log.

The stack trace is provided below as to show what happens every time a
logger.info() is executed in the PRServer.  Not only does it write to
the stream handler but it also tries to send the event to the main
event processor.

    self._notempty.acquire()
    self.queue.put(event)
    _ui_handlers[h].event.send(event)
    fire_ui_handlers(event, d)
    fire(record, None)
    self.emit(record)
    hdlr.handle(record)
    self.callHandlers(record)
    self.handle(record)
    self._log(INFO, msg, args, **kwargs)
    (self.dbfile, self.host, self.port, str(os.getpid())))
    self.work_forever()
    pid = self.daemonize()
    self.prserv.start()
    singleton.start()
    self.prhost = prserv.serv.auto_start(self.data)
    cooker.pre_serve()
    bb.cooker.server_main(self.cooker, self.main)
    self.run()
    code = process_obj._bootstrap()
    self._popen = Popen(self)
    self.serverImpl.start()
    server.detach()
    server = start_server(servermodule, configParams, configuration)
    ret = main()

It was never intended for the PRServer to send its logs anywhere but
its own log file.  The event processing is an artifact of how the
PRServer was forked and it inherits the event log handlers.  The
simple fix is to clean up and purge all the log handlers after the
fork() but before doing any of the typical PRServer work or logging.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: use a split/filter/rejoin for _remove

This is more idiomatic, and from the limited performance testing I did, is
faster as well. See https://gist.github.com/kergoth/6360248 for the naive
benchmark.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: allow removal of multiple words at once with _remove

    FOO = "foo bar baz"
    FOO_remove = "foo baz"

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
serv.py: Fix regression from 972bc43e6d5b

commit 972bc43e6d5b1207b944b3baa8f9805adb35dda7 (serv.py: Fix hang
when spawned dynamically with bitbake) introduced a regression,
because the wrong patch was submitted.  The syntax was incorrect in
the original patch.  The logger iterator must be used with a call to
getLogger().

[YOCTO #5059]

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv/db: Threading fixes

Enabling threading for the PRServer causes a number of issues. Firstly is
the obtuse error:

sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type

which is due to the class not being derived from object. See:
http://docs.python.org/2/library/sqlite3.html#registering-an-adapter-callable

Secondly, we want to enable multithreadded access to the database so we do this
when we open it. This opens the way up to multithreading the PR server.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>