scripts: Add checkpatch.pl from the QEMU project

OSv's coding style is very similar to that of QEMU.  The script probably
needs some work with our C++ specific coding style but we have to start
somewhere...

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
vfs: Unmount /proc to release refcnts pertained to root

/proc must be unmounted to release refcnts which pertains to the root
mountpoint, i.e. zfs.
It was preventing zfs_umount from releasing the mp dentries properly,
thus VOP_INACTIVE from being called on the respective vnodes.

Found the problem while dumping the mountpoint refcnts.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Add SCOPE_LOCK(mutex) macro

Add a macro SCOPE_LOCK(mutex) which locks the given mutex and unlocks
it when the scope ends (this uses RAII, so the mutex will correctly get
unlocked even when the scope is exited via return or exception).

This does the same as C++11's std::lock_guard, but far less verbose:
To use std::lock_guard with a mutex m, one nees to do something like
std::lock_guard<mutex> guard(m);
where the mutex's type needs to be repeated, and a name needs to be
invented for the guard which will likely not be used again. This
macro makes these things unnecessary, and one just writes
SCOPE_LOCK(m);

Note that WITH_LOCK(m) { ... } should usually be preferred over SCOPE_LOCK.
However, SCOPE_LOCK can come in handy in some cases, for example adding a
lock to a function without reindenting it.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Fix loader.cc parallel running bug

When running a command in the background, do_main_thread() passes the
command line in a std::vector pointer to a new pthread. Unfortunately,
soon afterwards the vector can go out of scope and the result is a
crash. Fix this oversight.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Elf: Serialize shared-object load and unload

Our current dynamic-linker code (elf.cc) is not thread safe, and all sort
of disasters can happen if shared objects are loaded, unloaded and/or used
concurrently. This and the following patches solve this problem in stages:

The first stage, in this patch, is to protect concurrent shared-library
loads and unloads. (if the dynamic linker is also in use concurrently,
this will still cause problems, and will be solved in the next patches).

Library load and unload use a bunch of shared data without protection,
so concurrency can cause disaster. For example, two concurrent loads can
pick the same address to map the objects in. We solve this by using a mutex
to ensure only one shared object is loaded or unloaded at a time.

Instead of this coarse-grain locking, we could have used finer-grained
locks to allow several library loads to proceed in parallel, protecting
just the actual shared data. However the benefits will be very small
because with demand-paged file mmaps, "loading" a library just sets up
the memory map, very quickly, and the object will only be actually read
from disk later, when its pages get used.

Fixes #175.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Elf: Fix rare load/unload race

The previous patch fixed most of the concurrent shared-object load/unload
bugs except one: We still have a rare race between loading and unloading the
*same* library.

In other words, it is possible that just when a library's reference count
went down to 0, and its destruction will promptly start (the mutex is not
yet taken), get_library() gets called to load the same library. We cannot
return the already-loaded one (because it is in the process of being
destructed). So in that case we need to ignore the library we found,
and load it again. Luckily, std::weak_ptr's notion of "expired" shared
pointers (that already went down to 0, and can't be lock()ed) and its
atomic implementation thereof, makes fixing this problem easy and natural.

The existing code already had an assert() in place to protect against this
race (and it was so rare we never actually saw it), but this patch should
fix this race once and for all.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Elf: Fix shared-object load concurrent with dynamic linker use

This patch addresses the bugs of *use* of the dynamic linker - looking
up symbols or iterating the list of loaded objects - in parallel with new
libraries being loaded with get_library().

The underlying problem is that we have an unprotected "_modules" vector
of loaded objects, which we need to iterate to look up symbols, but this
list of modules can change when a new shared object is loaded.

We decided *not* to solve this problem by using the same mutex protecting
object load/unload: _mutex. That would make boot slower, as threads using
new symbols are blocked just because another thread is concurrently loading
some unrelated shared object (not a big problem with demand-paged file
mmaps). Using a mutex can also cause deadlocks in the leak detector,
because of lock order reversal between malloc's and elf'c mutexes: malloc()
takes a lock first and then backtrace() will take elf's lock, and on the
other hand elf can take its lock and then call malloc taking malloc's lock.

Instead, this patch uses RCU to allow lock-free reading of the modules
list. As in RCU, writing (adding or removing an object from the list)
manufactures a new list, defering the freeing of the old one, allowing
reads to continue using the old object list.

Note that after this patch, concurrent lookups and get_library() will
work correctly, but concurrent lookups and object *unload* still will
still not be correct because we need to defer an object's unloading from
memory while lookups are in progress. This will be solved in a following
patch.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Elf: Refactor shared-object deleter

In the current code, when a shared-object's reference count drops to zero,
shared_ptr<object> calls delete on this object, which calls the object's
destructor, which (see file::~file) calls program::remove_object() to
remove this object from the program's _files list.

This order was always awkward and unexpected, and I even had a comment
in ~file noting that "we don't delete(ef) here - the contrary, delete(ef)
calls us".

But for the next patch, we can no longer live with this order, which is
not just awkward but also wrong: In the next patch we'll want a shared
object to be immediately removed from the program (so it is no longer
found when resolving symbols, etc.), but the actual deletion of this
object to be delayed till a more convenient time. For this to work, we
cannot have the deletion of the object start before its removal from the
program: We need the opposite - the removal of the object from the program
will delete the object.

Luckily, it's easy to do this properly using a shared_ptr deleter lambda.
This tells shared_ptr to call remove_object(), not delete, when the
reference count drops to zero.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Elf: Fix shared-object unload concurrent with dynamic linker use

After the above patches, one race remains in the dynamic linker: If an
object is *unloaded* while some symbol resolution or object iteration
(dl_iterate_phdr) is in progress, the function in progress may reach
this object after it is already unmapped from memory, and crash.

Therefore, we need to delay unmapping of objects while any object
iteration is going on. We need to allow the object to be deleted from
the _modules and _files list (so that new calls will not find it) but
temporarily delay the actual freeing of the object's memory.

The cleanest way to achieve this would have been to increment each
modules' reference in the RCU section of modules_get(), so they won't
get deleted while still in use. However, this will signficantly slow down
users like backtrace() with dozens of atomic operations. So we chose
a different solution: keep a counter _modules_delete_disable, which
when non-zero causes all module deletion to be delayed until the counter
drops back to zero. with_modules() now only needs to increment this
single counter, not every separate module.

Fixes #176.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Elf: Fix also _module_index_list

Fix also concurrent use of _module_index_list (for the per-module TLS).
Use a new mutex _module_index_list mutex to protect it. We could
probably have done something with the RCU instead, but just adding a
new mutex is a lot easier.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
runtime: Support format specifies in abort()

Add format specifier support to abort() to make it easier to produce
useful error messages.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
jvm_balloon: handle explicit unmapping case

The JVM may unmap certain areas of the heap completely, which was confirmed by
code inspection by Gleb. In that case, the current balloon code will break.

This is because we were deleting the vma from finish_move(), and recreating the
old mapping implicitly in the process. With this new patch, the tear down of
the jvm balloon mapping is done by a separate function. Unmapping or evacuating
the region won't trigger it.

It still needs to communicate to the balloon code that this address is out of
the balloons list. We do that by calling the page fault handler with an empty
frame. jvm_balloon_fault will is patched to interpret an empty frame correctly.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
run.py: introduce --vnc option

It will allow the user to specify the vnc port number because
the port "1" is often occupied by other applications.

Signed-off-by: Zhi Yong Wu <zwu.kernel@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
java: print errors on stderr rather than debug ring.

Currently debug messages are not printed on console by default.
java: print exception thrown from FindClass

env->FindClass() triggers class initialization which may throw
exceptions. Instead of printing misleading information that the class
was not found we should print the exception stack-trace.
java: extract abstraction of isolated application

We aim to support running multiple isolated Java applications in one
JVM. Some work has already been done to isolate system class
loaders. There is much more to it than that. Isolated applications
(aka Contexts) do not map 1-1 to class loaders. One context may have
many different class loaders.

This change extracts context-specific logic to separate classes as a
base for further additions.
java: introduce utilities for lazy initialization

I need static-initialization-like semantics in several places.
This abstraction makes it easier.
java: create standard java module structure in runjava module

There is a convention to organize java modules in the following way:

src/main/java - source root for production code
src/test/java - source root for test code
java: use maven to build runjava module

It will allow to use automatic dependency management.

It slows down the build a bit. Incremental build takes 3 seconds
longer than previously.

First build takes longer due to downloading of maven artifacts.
This is once per machine.
java: implement isolation of java.util.logging managers

The j.u.l framework allows only one log manager to exist. To isolate
logging configurations we need to install our own log manager which
delegates to context-local log managers.

See #172.

In order to have a fully isolated logging config system properties
need to also be isolated. This will come in the following patches.