bitbake/event.py: UIhandler filter should work without a mask

The default for the mask will be * (all the handlers)

Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob: add event handlers filtering in Hob

Create the _evt_list for hob; it is longer than the knotty
uses because it handles more events.

Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
hob: fixes for image combo box

When an image from scratch is selected, and recipes parsing
is canceled, the image shown by the combo box isn't correct.

[YOCTO #5000]
Signed-off-by: Cristiana Voicu <cristiana.voicu@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
serv.py: Fix hang when spawned dynamically with bitbake

The PRServer has the possibility to hang indefinitely blocking on a
semaphore processing a xmlrpc request to send an event back to the
main bitbake instance.  This was observed during a "bitbake -e" on a
heavily loaded machine and the main bitbake instance and cooker exited
before the PRServer emitted its first log.

The stack trace is provided below as to show what happens every time a
logger.info() is executed in the PRServer.  Not only does it write to
the stream handler but it also tries to send the event to the main
event processor.

    self._notempty.acquire()
    self.queue.put(event)
    _ui_handlers[h].event.send(event)
    fire_ui_handlers(event, d)
    fire(record, None)
    self.emit(record)
    hdlr.handle(record)
    self.callHandlers(record)
    self.handle(record)
    self._log(INFO, msg, args, **kwargs)
    (self.dbfile, self.host, self.port, str(os.getpid())))
    self.work_forever()
    pid = self.daemonize()
    self.prserv.start()
    singleton.start()
    self.prhost = prserv.serv.auto_start(self.data)
    cooker.pre_serve()
    bb.cooker.server_main(self.cooker, self.main)
    self.run()
    code = process_obj._bootstrap()
    self._popen = Popen(self)
    self.serverImpl.start()
    server.detach()
    server = start_server(servermodule, configParams, configuration)
    ret = main()

It was never intended for the PRServer to send its logs anywhere but
its own log file.  The event processing is an artifact of how the
PRServer was forked and it inherits the event log handlers.  The
simple fix is to clean up and purge all the log handlers after the
fork() but before doing any of the typical PRServer work or logging.

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: use a split/filter/rejoin for _remove

This is more idiomatic, and from the limited performance testing I did, is
faster as well. See https://gist.github.com/kergoth/6360248 for the naive
benchmark.

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
data_smart: allow removal of multiple words at once with _remove

    FOO = "foo bar baz"
    FOO_remove = "foo baz"

Signed-off-by: Christopher Larson <chris_larson@mentor.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
serv.py: Fix regression from 972bc43e6d5b

commit 972bc43e6d5b1207b944b3baa8f9805adb35dda7 (serv.py: Fix hang
when spawned dynamically with bitbake) introduced a regression,
because the wrong patch was submitted.  The syntax was incorrect in
the original patch.  The logger iterator must be used with a call to
getLogger().

[YOCTO #5059]

Signed-off-by: Jason Wessel <jason.wessel@windriver.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv/db: Threading fixes

Enabling threading for the PRServer causes a number of issues. Firstly is
the obtuse error:

sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type

which is due to the class not being derived from object. See:
http://docs.python.org/2/library/sqlite3.html#registering-an-adapter-callable

Secondly, we want to enable multithreadded access to the database so we do this
when we open it. This opens the way up to multithreading the PR server.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv/serv: Multithread the server

This makes the PR server multithreaded and able to handle multiple connections
at once which means its no longer a build bottle neck when serving one connection
at a time. I've experimented and database connection for each thread seems to
cause the least issues, pushing the contention for sqllite to handle itself.

This means moving the db/table connection code into the actual function methods.
It doesn't abstract well as a function since we need the db object around for
the lifetime of the function as well as the table else we lose the connection.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cookerdata: Set TOPDIR when using bblayers.conf

By definition, bblayers.conf is at the top of the build tree. We'd like
to support running bitbake anywhere within that build tree but TOPDIR
gets set to wherever cwd is. Change the code to reset TOPDIR
to the top of the build directory.

This shouldn't break anything but does make the system more usable.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
cookerdata: Allow bblayers.conf to be found using BBPATH

It should be possible to run a build anywhere on the filesystem and have
bitbake find the correct build directory if its set somehow. The BBPATH
variable makes perfect sense for this usage. Therefore use any available
value of BBPATH to search for conf/bblayers.conf before walking the parent
directory structure.

This restores the option of being able to run bitbake from anywhere if
the user has set things up to operate in that environment.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
server/xmlrpc: Increase timeout to 60s

This is a better value that the earlier infinite timeout yet still
allows for servers with high loads. It does mean the bitbake process
can hang at exit for the timeout period but that should never happen
and only happened for me in some test cases which wouldn't happen
in normal use.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv: Allow 'table is locked' matching for retry loop

Try and avoid errors like "ERROR: database table is locked: PRMAIN_nohist"
by retrying if we see the string "is locked".

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
server/process, server/xmlrpc, runqueue: Use select.select() on fds, not time.sleep()

The existing backend server implementations were inefficient since they
were sleeping for the full length of the timeouts rather than being woken when
there was data ready for them. It was assumed they would wake and perhaps did
when we forked processes directory but that is no longer the case.

This updates both the process and xmlrpc backends to wait using select(). This
does mean we need to pass the file descriptors to wait on from the internals
who know which these file descriptors are but this is a logical improvement.

Tests of a pathaolgical load on the process server of ~420 rapid tasks
executed on a server with BB_NUMBER_THREAD=48  went from a wall clock
measurement of the overall command execution time of 75s to a much more
reasonable 24s.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
serv/db: Fix looping upon database locked issues

If the database is locked we will get an immediate error indicating so,
there is no retry timeout. The looping code is therefore useless, the loop
count is near instantly exceeded.

Using a time based retry means we can wait a sensible time, then gracefully
exit.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
serv/db: Take an excluside lock on the database

We only support one server using the database at a time so take an exclusive
lock and avoid later lock overhead.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
serv/db: Don't use BEGIN/COMMIT

Since we don't support using multiple servers on the same database file,
don't use the BEGIN/COMMIT syntax and allow writes to the database
to work ~100 times faster with no transaction locking.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
build: Fix profile file names

Using the basename of the .bb file is not unique, for example xxx-native
and xxx can overwrite each other. If this happens whilst running, you can
get odd backtraces as one file is parsed as another tries to write out
new data.

Avoid issues by using PN for the output filename instead.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
prserv/serv: Settle on two threads for optimal performance

Using the threading mixin class resulted in large amounts of memory
being used by the PR server for no good reason. Using a receiver thread
and a thread to do the actual database operations on a single connection
gives the same performance with a much saner memory overhead so
switch to this.

Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>
server/xmlrpc: stop server on client exit

If the server only expects a single client to connect (i.e.
no bind parameter set, so there is no way for the
clients to get the server port), stop the server after
the first client exits.

Signed-off-by: Alexandru DAMIAN <alexandru.damian@intel.com>
Signed-off-by: Richard Purdie <richard.purdie@linuxfoundation.org>