balloon: use a separate thread the for the balloon

Synchronously ballooning unfortunately won't work: as usual the garbage
collector can run in order to release memory for the balloon, and then we will
block.

Since we can't make the balloon worker synchronous, we set its priority to a
really high number, that would guarantee that Java threads won't run before it.
This will help in really fast allocation situations. Durin those, between the
notification and the time this thread gets to run, the JVM may have very well
exhausted all the memory remaining.

Notice that this value is guaranteed to be lower than any JVM created thread
ever will get. This is since for JVM threads, priority will only be set through
the setpriority interface, wich will yield 0.2 for its maximum prio.
jvm_balloon: fix braino

small logic invertion: it is when we *do* copy that we should break. Was not important
up to now because we were always executing this loop once. It is important now because
we will execute it multiple times.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
balloon: close the balloons earlier if the regions intersect

Instead of just looking at which region our start address falls at, we can
iterate through all the vmas that are affected by us installing a new vma, and
then close the region earlier.

This design has one major big advantage: split() will never happen, and if that
is the case, the balloon becomes a lot more predictable - which at this point
is a blow of sanity in the whole thing

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
balloon: fix move to self with partial copy

A similar bug that has been already fixed with move-to-self for the normal
balloon case also happens with partial copies. Fix them here, as early in the
code path as we can (that last part, to avoid useless spent cycles). Reminding
again, "move to self" happens not when the JVM move the object to the same
address - which would be stupid, but to an address that is close enough to the
old address to fall in the same mapping.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
mempool: transform reclaimer thread into a normal thread

The balloon is no longer synchronous, and now lives in its own
thread. We used to have the requirement that the reclaimer needed
to have a full pthread API, but that is no longer. Make it now a
normal thread.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
reclaimer: add tracepoint for waiters

Useful for debugging, specially for the backtrace

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
mempool.cc: print the amount of memory available when we oom

We consider ourselves to be "out of memory" when an allocation can not be
satisfied, not when we reach 0 free memory. If we are heavily fragmented, those
two situations can be quite departed from each other. I have never printed the
amount of free memory at oom because abort() could not handle a format. Now it
can.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
reclaimer: wipe out conditionals

One of the reasons we are being overly aggressive with ZFS reclaim is that once
we reach the reclaimer, unless we take ourselves away from the pressure
situation, we will keep reclaiming until we do. That happens even if we don't
need any extra memory. This is particularly problematic given the way ZFS
reclaimer is designed: if it is called twice in a row, it changes its mode of
operation from conservative (ARC_RECLAIM_CONS) to aggressive
(ARC_RECLAIM_AGGR), and will shrink the cache to the current cache usage.

Since we will wake the reclaimer every time we have reason to believe memory is
under pressure, there is no need for the reclaimer itself to keep looping.

After this patch is applied, the reclaimer is still called sequentially if the
allocation rate is high (because we will wake it many times), but will be left
alone if it isn't.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
reclaimer: only wake reclaimer when memory is used by OSv itself

The current waking criteria - to wake up the reclaimer when memory is overall
short - won't really work when we have ballooning enabled: in that situation,
the whole of memory will most likely be taken by the heap. This means that we
will constantly be bellow the threshold and will reclaim at every opportunity.

OSv memory usage won't grow, and we won't be placing memory when it is most
needed.  What we really need is to keep waking the reclaimer when memory usage
caused by OSv is high, but for the balloon policy itself we will need our own
calls.

stats::jvm_heap() will return 0 when we have no balloon, therefore that case
will keep seeing no differences.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
bsd_shrinkers: create specialized ARC shrinker

It will give us more freedom to apply ARC shrinking strategies. It is
particularly important given the fact that ARC has two modes of shrinking
(aggressive and conservative) and we may not always want to trigger a policy
change.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
shrinkers: allow soft and hard mode

There are some shrinkers, like the ARC, that may become very agressive when
called more than once in a row. When we have no waiters, we can be more relaxed
and just go for a best effort shrinking.

This patch introduce a soft mode shrinking that will do just that. In the
future, we may introduce another threshold smaller than the minimum in which we
can go in hard mode even before we have waiters. For now, this will do.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
mmu: remove lock guard in balloon_fault

The fault method itself is always called with the lock held. Locking
it again will only slow us down.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
bsd: also use jvm figures in kmem_used calculations

We are defined kmem_used() as total - free. However, as the name implies, this
field is supposed to report on the amount of kernel memory used, not total phys
memory.

Granted, this situation is trickier for us to determine, because we're running
everything together. But in some situations, this can lead to an interesting
performance deterioration: if the JVM touches all the memory for the heap, this
will lead to a very large memory usage, even though the ARC itself is using very
little of it. The ARC will then shrink, and will never have the chance of grow
to the sizes it should. Including the jvm_heap figures will allow the ARC to
have a better idea about what happens.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
OSvMonitor: change criteria for removing balloons

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
jvm_balloon: more tracing functions

Let's track not only move, but creation and destruction

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
zfs: throttle ZFS allocations during balloon

When we create a new balloon, memory will soon be available. However, until
we actually create it, high rate ARC users will continue to fill the ARC at
will, which can easily lead to memory exhaustion.

Blocking allocations until the balloon is created is not an option: in order to
balloon, we need to allocate memory, and the source of this memory is quite
unpredictable: backing heap, stack, and even file, if an object destruction
needs to flush something out.

Luckily, the ARC has a mechanism to avoid growing the cache. If the function
arc_evict_needed() returns true, it will free buffers to give place to new ones
instead of growing them.

The most natural way of making that function return true would be to implement
the function vm_paging_needed(), which is already there in generic code and
would require 0 ARC changes. However, doing this would imply that
arc_reclaim_needed() would now return true as well. This would not only evict
some buffers, but actually put the ARC in aggressive mode (Especially because
the allocations will keep coming)

The solution I've chosen was to implement a new function in the ARC - wrapped
in an ifdef so we don't lose track of it - that will allow us to communicate
that we should throttle new buffers due to a VM request (in our case, because
we are waiting for the balloon).

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
jvm_balloon: correctly calculate total heap

the function totalMemory won't necessarily give us what we want. It will return
total committed memory, not total memory. The correct function is maxMemory.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
zfs: give IO threads access to emergency pools

All the IO issue threads are given access to emergency pools. Although I would
prefer to cut down to only the write and sync ones, this should do because even
the read ones, are only responsible for actually carrying out the IO (all the
IO setup happens in the normal codepath). I do it this way because it is hard
to differentiate between them.

Additionally, the transaction sync thread is also given access.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
balloon: return memory voluntarily

There are situation in which the JVM allocation rate is way too high. To
account for that, it is better to also return memory voluntarily when there is
memory available.

If the memory is just made available, we won't do anything. But after the
reclaimer is called, it might be that the ARC cache is flushed and memory
is now available. If the memory now available is more than the balloon + a
threshold, we will give it back.

Since we assume that all the memory available for the heap will eventually be
consumed, the very fact that the reclaimer itself was called is a sign that
we are using too much kernel memory.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
reclaimer: do not go hard if we're throttling

In case we are throttling, try to serve memory from the non-aggressive
shrinker before going all-in. In particular, the current waiter may be
allocating memory in behalf of the balloon (for instance, stack), so it
is a bad idea to block. We may have more memory soon enough.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>