reclaimer: implement a stronger data structure for waking sleepers

After Avi's last comments, here is a new take for the reclaimer's wake queue.
It does not abuse semaphore's in any way, so there is no need to extend it.
(After this patch is applied, the semaphore extension patch can be reverted)

In this new approach, we will not only check for free memory as a scalar, but
will instead transverse the remaining page ranges (notice that in case of
fragmentation this can be a *large* set) and try to find a range that would
allow our allocation to happen.

Also, because we are implementing both the wait and the "post" methods, the
waking of the reclaimer happens atomically and the problem that Tomek described
(wake before waiters is set) no longer exists.

Finally, another advantage of this approach is that we no longer need to keep
track of the amount of memory we have freed when making OOM decisions: if we
can wake someone up, we can continue. Otherwise, we OOM. This will be specially
helpful for the FS shrinkers, since they may not always be able to synchronously
return a meaningful number accurately (due to, for instance, writeback).

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
jvm_balloon: better define boundaries for the balloon.

Right now, we are starting the jvm with the maximum amount of memory we can.
This is, however, terribly bad. The reason is that if that if we go to reclaim
during a heap allocation, this will easily crash the system. That will be just
for granted if that allocation turns out to be done on behalf of the GC. In
that case, no other Java threads will make progress, meaning the balloon cannot
be activated.

Even if the balloon is no longer part of the shrinker infrastructure, it may
still be a problem, because we may need to use it to deflate memory. The best
solution is to just limit heap usage to 90 % of the available memory, which is
the reclaim criteria.

Also, it is pointless to activate the balloon for very low memory conditions.
So if our total memory is less than 1Gb, just leave it alone.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
java: heuristically determine the heap regions at startup

We can determine which mappings from the JVM are heap areas in a more
robust way by flipping a flag when we start JVM creation, and flipping
it back when we are finished. Doing this early is more robust and should
work better with a bigger set of GCs.

Although this is a heuristic, Calle agrees this is safe, and it has
been tested and confirmed to work with the following options:

    -XX:+UseSerialGC
    -XX:+UseG1GC
    -XX:+UseParallelGC
    -XX:+UseParallelOldGC
    -XX:+UseParNewGC
    -XX:+UseConcMarkSweepGC
    -XX:+UseConcMarkSweepGC

which pretty much covers all relevant GCs

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
jvm_balloon: remove the probe

New method does not rely on physical memory accounting, so the probe
is useless.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
jvm balloon: does not take the balloon lock during fault

The lock itself is no longer needed - it was in the past before the balloon was
directly put in the vma. But besides that, it is leading to subtle deadlocks:
let's say a new balloon is being inserted into the list, which needs to be done
with lock held. If Garbage Collection kicks in at this very same time and
happens to move a balloon, it will stop in the lock. This means that all java
threads will stop, waiting for the GC to finish. All it takes is a single JNI
call with the lock taken to achieve a deadlock. In particular, deleting a balloon
involves destroying a reference, so that is enough to trigger it.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
jvm_balloon: handle partial writes

There are situations during which some Garbage Collectors will copy a large
object in parallel, using various threads, each being responsible for a part of
the object. In particular, that easily happens under high memory pressure for
the Parallel and ParallelOld collectors.

If such copy happens, the simple balloon move algorithm will break. However,
because offset 'x' in the source will always be copied to offset 'x' in the
destination, we can still calculate the final destination object.  The problem
is that we cannot open the new balloon yet. Since the JVM believes it is
copying only a part of the object, the destination may (and usually will)
contain valid objects, that need to be themselves moved somewhere else before
we can install our object there.

Also, we can't close the object fully when someone writes to it: because a part
of the object is now already freed, the JVM may and will go ahead and copy
another object to this location.

This is a quite tricky problem, but the patch bellow solves it by keeping the
source balloon open and counting the number of partial bytes copied. It makes
the assumption that the JVM needs to always copy the whole object to a different
location, meaning that when it does, the balloon can be closed.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
map_jvm: update the balloon object on address reuse

If we are reusing an address, it might be that we are doing for the same
balloon - move-to-self scenario, or it might be that simply there is another
balloon that is moved to the address where previously another balloon was held,
but before we had the chance to close it. In the later case, the balloon object
- which is used to compute offsets in the object is wrong, and we will copy
things to unpredictable locations. Need to update it correctly.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
zfs: add ability to control amount to be reclaimed

Currently, although arc_evict() can be told a size up until it should reclaim,
the high level reclaimer is always too aggressive, reclaiming up to a
parametrized value. There are situations in which we would just like to reclaim
enough to make room for something. For those cases, I am introducing here a
function that allow us to inform ZFS what is the size we need to see evicted
from the caches.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
balloon: no longer make it part of the shrinker infrastructure.

The current ballooning mechanism has a very serious design flaw, that became
obvious once the tests with it advanced: if we wait until memory is short to
release memory, it may be that the JVM itself may block. In particular, the
blocking thread can be running the Garbage Collector. More likely, it can
start running the GC to open up space for an object so big as the balloon.

I don't believe this is something we should fix with a quick hack, and I am
then pushing for a redesign: The solution is to use a reservation system, that
works by ballooning beforehand when the amount of reserved memory goes too low.

This system works by noting that when we balloon, we are effectively limiting
the maximum amount of memory java will ever use. We already have accounts of
both the amount of free memory and the amount of memory allocated to the heap.
With those figures, we can easily find out when the amount of used memory +
a full heap would go over the amount of available memory. When that happens,
we balloon.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
mmu: return vma size instead of evacuated pages

As we move towards a reservation system, the actual memory that we have freed
matters little. Return instead the size of the vma that was passed to OSv

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
balloon: no longer use recent statistics

We are moving the balloon out of the reclaimer path, so those does not
really make a lot of sense.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
balloon: use a separate thread the for the balloon

Synchronously ballooning unfortunately won't work: as usual the garbage
collector can run in order to release memory for the balloon, and then we will
block.

Since we can't make the balloon worker synchronous, we set its priority to a
really high number, that would guarantee that Java threads won't run before it.
This will help in really fast allocation situations. Durin those, between the
notification and the time this thread gets to run, the JVM may have very well
exhausted all the memory remaining.

Notice that this value is guaranteed to be lower than any JVM created thread
ever will get. This is since for JVM threads, priority will only be set through
the setpriority interface, wich will yield 0.2 for its maximum prio.
jvm_balloon: fix braino

small logic invertion: it is when we *do* copy that we should break. Was not important
up to now because we were always executing this loop once. It is important now because
we will execute it multiple times.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
balloon: close the balloons earlier if the regions intersect

Instead of just looking at which region our start address falls at, we can
iterate through all the vmas that are affected by us installing a new vma, and
then close the region earlier.

This design has one major big advantage: split() will never happen, and if that
is the case, the balloon becomes a lot more predictable - which at this point
is a blow of sanity in the whole thing

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
balloon: fix move to self with partial copy

A similar bug that has been already fixed with move-to-self for the normal
balloon case also happens with partial copies. Fix them here, as early in the
code path as we can (that last part, to avoid useless spent cycles). Reminding
again, "move to self" happens not when the JVM move the object to the same
address - which would be stupid, but to an address that is close enough to the
old address to fall in the same mapping.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
mempool: transform reclaimer thread into a normal thread

The balloon is no longer synchronous, and now lives in its own
thread. We used to have the requirement that the reclaimer needed
to have a full pthread API, but that is no longer. Make it now a
normal thread.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
reclaimer: add tracepoint for waiters

Useful for debugging, specially for the backtrace

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
mempool.cc: print the amount of memory available when we oom

We consider ourselves to be "out of memory" when an allocation can not be
satisfied, not when we reach 0 free memory. If we are heavily fragmented, those
two situations can be quite departed from each other. I have never printed the
amount of free memory at oom because abort() could not handle a format. Now it
can.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
reclaimer: wipe out conditionals

One of the reasons we are being overly aggressive with ZFS reclaim is that once
we reach the reclaimer, unless we take ourselves away from the pressure
situation, we will keep reclaiming until we do. That happens even if we don't
need any extra memory. This is particularly problematic given the way ZFS
reclaimer is designed: if it is called twice in a row, it changes its mode of
operation from conservative (ARC_RECLAIM_CONS) to aggressive
(ARC_RECLAIM_AGGR), and will shrink the cache to the current cache usage.

Since we will wake the reclaimer every time we have reason to believe memory is
under pressure, there is no need for the reclaimer itself to keep looping.

After this patch is applied, the reclaimer is still called sequentially if the
allocation rate is high (because we will wake it many times), but will be left
alone if it isn't.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
reclaimer: only wake reclaimer when memory is used by OSv itself

The current waking criteria - to wake up the reclaimer when memory is overall
short - won't really work when we have ballooning enabled: in that situation,
the whole of memory will most likely be taken by the heap. This means that we
will constantly be bellow the threshold and will reclaim at every opportunity.

OSv memory usage won't grow, and we won't be placing memory when it is most
needed.  What we really need is to keep waking the reclaimer when memory usage
caused by OSv is high, but for the balloon policy itself we will need our own
calls.

stats::jvm_heap() will return 0 when we have no balloon, therefore that case
will keep seeing no differences.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>