Rename mutex_*() functions to __mutex_*().

To avoid symbol clashes with /usr/include/synch.h on Solaris 11.1.

Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
iv_event: Get rid of iv_tls usage.

As iv_event has become one of the core ivykis modules, integrate its
per-thread state directly into struct iv_state instead of making it
use iv_tls.

Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
test/avl: Reduce test AVL tree size.

So that the testsuite doesn't take forever on slower machines.

Reported-by: Christian T. Steigies <cts@debian.org>
Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
Turn iv_validate_now() into a no-op.

When support for pthread_[gs]etspecific() was added, 'iv_now' was
changed from being a __thread variable to being a macro dereferencing
the pointer return value of a call to a function in the ivykis core
(commit 5eed214c3e22, "iv_timer: Move per-thread state to struct
iv_state."), with the goal of moving knowledge of how thread-local
state is stored out of the iv.h public header file.

However, this changed the 'call iv_validate_now() / access iv_now'
sequence of operations for accessing the monotonic clock from a
single function call and a thread-local value reference into a
sequence of two function calls, where one function call would be
enough to do the job.

This commit changes iv_validate_now() into an empty inline function,
and adds the clock validation logic to the function that returns the
address of iv_now (__iv_now_location()), so that all code compiled
against a newer ivykis version will only do a single function call to
return the current value of the monotonic clock.

To preserve API compatibility, iv_validate_now() will continue to
exist as an inline function for as long as we want to keep API/ABI
compatibility (likely forever), and the requirement for ivykis users
to call iv_validate_now() before referencing iv_now has not been
dropped by this commit, to ensure ivykis users will still work
properly when built and linked against older ivykis versions.

To preserve ABI compatibility for code compiled against an older
ivykis version and dynamically linked against a newer ivykis version,
the old version of iv_validate_now() will continue to be provided as
a symbol exported from libivykis.so.

To prevent ABI problems for code compiled against a newer ivykis
version and dynamically linked against an older ivykis version, the
new version of __iv_now_location() is given a different name, to
prevent code compiled against newer ivykis versions which cause
references to 'iv_now' to validate it if it's out of date from
ending up with a version of __iv_now_location() that doesn't perform
this validation.

Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
iv_fd: Let ->poll() return whether a timeout could have occurred.

After having polled for new events, poll methods can sometimes deduce
that the specified timeout cannot possibly have expired yet, and this
information can be used to avoid checking for expired timers on return
(which involves a potentially costly call to clock_gettime() or
gettimeofday() that we would gladly avoid).

This commit makes iv_fd_poll_method::poll() return an int, where zero
means "the specified timeout has definitely not expired yet", and a
nonzero value means "the timeout may or may not have expired yet".

Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
iv_fd: Allow timeout handling optimizations in poll methods.

All file descriptor polling methods that ivykis supports have timeout
arguments to their respective polling system calls -- epoll_wait(2),
poll(2) and poll(7d) have a relative milliseconds argument, and
kevent(2), port_getn(3C) and ppoll(2) have a relative struct timespec
pointer argument.

When calling into the kernel to wait for file descriptor activity to
occur, ivykis determines the timeout to be used for that system call
by finding the registered iv_timer with the soonest expiring timeout,
and passing in that timeout to the poll system call for the polling
method currently in use, so that that system call will return when
there is file descriptor activity or when the first iv_timer to be
expiring has expired.

One opportunity for improvement on this approach has to do with the
fact that on most operating systems, there appears to be a nontrivial
cost associated with setting up and tearing down a kernel timer on
every entry to and exit from a poll system call (which is what happens
if you pass in a sub-infinite timeout to such a system call), and that
some of those operating systems provide a way of setting up a kernel
timer that can be kept running across system calls.

Even on operating systems where setting up and tearing down kernel
timers isn't all that costly, it can be advantageous to set up a
separate kernel timer corresponding to the soonest expiring iv_timer,
as that will give us explicit notification of when that timer expires,
which can let us avoid calling clock_gettime() on every poll system
call return to check whether the current time is greater than or equal
to the soonest expiring iv_timer's timeout yet.

This commit adds two new poll method methods, ->set_poll_timeout()
and ->clear_poll_timeout():

* ->set_poll_timeout(), when implemented, will be called by the ivykis
  core when it wants to configure an explicit timeout, which when it
  expires must cause a subsequent or in-progress call to ->poll() to
  return (with a nonzero value), even if that call to ->poll() was done
  with a NULL (infinite) timeout.

* ->clear_poll_timeout() will be called if an explicit timeout was
  previously configured which has now been superseded, for example if
  an iv_timer with a timeout earlier than the currently registered
  explicit timeout is registered before the explicit timeout expires.

Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
epoll: Implement ->set_poll_timeout() using timerfd_create(2).

Without this commit, iv_event_raw_bench produces the following
performance figures (median values of five runs) on a virtual machine
running CentOS 6.4 (which supports timerfd_create(2)):

	signal: 13762361 in 5000018382 nsec => 2752462/sec
	timer: 8747421 in 5000005528 nsec => 1749482/sec

This commit implements ->{set,clear}_poll_timeout() for the epoll poll
method using timerfd_create(2), allowing offloading of timeout handling
from epoll_wait(2) to a timerfd so that we can call epoll_wait(2) with
timeout values of -1 under the right conditions, which avoids setting
up and tearing down a kernel timer on every epoll_wait(2) call.  As a
side effect, this commit also allows avoiding calls to clock_gettime()
as long as no timers are expiring and noone is referencing iv_now or
calling iv_validate_now().  With these changes, the results are:

	timer: 13750406 in 5000013792 nsec => 2750073/sec

In other words, this commit virtually eliminates all of the cost of
having a background timer running.

To see which of the two effects (timer offloading or clock_gettime()
avoidance) helps the most, I modified this commit to always call
clock_gettime() in the ivykis main loop to check for expired timers
even if we know that no timers can have expired yet, and that gives us:

	timer: 10274371 in 5000013266 nsec => 2054868/sec

Which means that the timer offloading gives an improvement of about
300k loops per second, and the clock_gettime() avoidance gives an
additional improvement on top of that of about 700k loops per second.

Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
kqueue: Implement ->set_poll_timeout() using EVFILT_TIMER.

Without this commit, iv_event_raw_bench produces the following
performance figures (median values of five runs) on a virtual machine
running FreeBSD 9.1:

	signal: 6262268 in 5010000550 nsec => 1249953/sec
	timer: 5249789 in 5000003980 nsec => 1049956/sec

This commit implements ->{set,clear}_poll_timeout() for the kqueue
poll method using EVFILT_TIMER, which like its epoll counterpart
commit allows offloading of timeouts from being per kevent(2) call
to persisting across kevent(2) calls, so that we can call kevent(2)
with NULL timeout values under the right conditions, and it also
allows avoiding calls to clock_gettime() as long as no timers are
expiring and noone is referencing iv_now or calling iv_validate_now().
With these changes, the results are:

	timer: 6252079 in 5010009760 nsec => 1247917/sec

So as for its epoll counterpart, this commit virtually eliminates
all of the cost of having a background timer running.

And as with the epoll commit, I modified this commit to always call
clock_gettime() in the ivykis main loop to check for expired timers
even if we know that no timers can have expired yet, and that gives us:

	timer: 5379069 in 5010003750 nsec => 1073665/sec

Meaning that avoiding setting up and tearing down kernel timers makes
almost no performance difference, but being able to avoid frequent
clock_gettime() calls helps quite a bit.

Further modifications to use CLOCK_MONOTONIC for clock_gettime()
instead of CLOCK_MONOTONIC_FAST, as we did on FreeBSD before ivykis
0.38, gives the following numbers, illustrating how much faster the
_FAST variant really is, or actually, how much slower the non-FAST
variant really is:

	timer: 820683 in 5000016670 nsec => 164136/sec

Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>
port: Implement ->set_poll_timeout() using timer_create(3C).

Without this commit, iv_event_raw_bench produces the following
performance figures (median values of five runs) on a virtual machine
running Solaris 11.1:

	signal: 2584578 in 4996484168 nsec => 517279/sec
	timer: 2292693 in 5000012397 nsec => 458537/sec

This commit implements ->{set,clear}_poll_timeout() for the
port_create(3C) poll method using timer_create(3C), which allows
keeping a kernel timer running across port_getn(3C) calls, so that
we can call port_getn(3C) with NULL timeout values under the right
conditions, and it also allows avoiding calls to clock_gettime() as
long as no timers are expiring and noone is referencing iv_now or
calling iv_validate_now().  With these changes, the results are:

	timer: 2585753 in 5000023204 nsec => 517148/sec

So as for the corresponding epoll and kqueue changes, this commit
virtually eliminates all of the cost of having a background timer
running.

And as in the other two cases, I modified this commit to always call
clock_gettime() in the ivykis main loop to check for expired timers
even if we know that no timers can have expired yet, and that gives us:

	timer: 2309814 in 5000019122 nsec => 461961/sec

This means that the kernel timer setup and teardown overhead seems
minimal, but the savings obtained from being able to avoid doing a
lot of clock_gettime() calls are significant.

Signed-off-by: Lennert Buytenhek <buytenh@wantstofly.org>