gdb: improve 'osv zfs' command

This patch improves the command by adding useful info for debugging ZFS
in general, and also addresses some stylistic issues.

The new output is as follow:
(gdb) osv zfs
:: ZFS TUNABLES ::
	zil_replay_disable:       0
	zfs_nocacheflush:         0
	zfs_prefetch_disable:     0
	zfs_no_write_throttle:    0
	zfs_txg_timeout:          5
	zfs_write_limit_override: 0
	vdev_min_pending:         4
	vdev_max_pending:         10
:: ARC SIZES ::
	Actual ARC Size:        122905056
	Target size of ARC:     1341923840
	Min Target size of ARC: 167740480
	Max Target size of ARC: 1341923840
		Most Recently Used (MRU) size:   670961920 (50.00%)
		Most Frequently Used (MFU) size: 670961920 (50.00%)
:: ARC EFFICIENCY ::
Total ARC accesses: 42662
	ARC hits: 41615 (97.55%)
		ARC MRU hits: 12550 (30.16%)
			Ghost Hits: 0
		ARC MFU hits: 29045 (69.79%)
			Ghost Hits: 0
	ARC misses: 1047 (2.45%)
Prefetch workload ratio: 0.0097%
Prefetch total:          412
	Prefetch hits:   20
	Prefetch misses: 392
Total Hash elements: 1053
	Max Hash elements: 1053
	Hash collisions:   13
	Hash chains:       11

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Refactor timerfd.cc

In his review of timerfd.cc, Avi asked that I simplify the implementation
by having a single "timerfd" object (instead of two I had - timerfd_file
and timerfd_object), and by using a single mutex instead of the complex
combination of mutexes and atomic variable.

This new version indeed does this. It should be easier to understand this
code, and it is 30 lines shorter.

The performance of this code is slightly inferior to the previous one -
in particular poll() now locks and unlocks a mutex - but this should be
negligible in practice.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
config.json: convert tabs to spaces

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: support nested module dependencies

Currently importing module from a module definition would fail because
we cannot call import module with the same name (module.py)
recursively, __import__ will complain that we removed 'module' from
sys.modules.

There is a simple solution to this problem, we can use runpy.run_path()
which works like a charm.

In addition to this we cache loaded modules so that we don't
have to load the file twice.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: make presence of either manifests optional

No need to create an empty bootfs.manifest anymore.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: cleanup resolve.py

No functional changes, just renames to more adequate names.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: make java a separate OSv module

This patch makes java files are copied to the guest image only when
'java' modue is included.

Modules can pull it explicitly by stating:

  require('java')

or implicitly, by creating api.run_java() run configurations.
In future we could consider moving api.run_java() into a java
meta-module.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: introduce higer level API to aid manifest generation

When plain manifests are not enough this is a concise alternative
with improved expresiveness. It allows to declare exclude and include
patterns. It's python based.

Example:

  m = FileMap()
  m.add('${OSV_BUILD_PATH}/tests').to('/tests') \
     .include('**/*.so') \
     .exclude('host/**')

Declared mappings can be saved in manifest form or be subject
of further processing. To save in manifest format:

  save_as_manifest(m, 'my.manifest')

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: allow for file mapping declaration in module.py

If module has 'usr_files' or 'bootfs_files' declared then
their value will be interpreted as FileMaps and appended
to appropriate manifests.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
java: add all JRE files to the image by default

Fixes issue with JVM failing when started with a debugger
with the following message:

  NPT ERROR: Cannot find nptInitialize

Missing openjdk files in usr.manifest were a fertile source
of issues. This patch aims at making them less likely and
adding all files   except blacklisted files to the image.

This patch skips two files from JRE which are broken links
and inclusion of which would cause manifest upload failure:
 - jre/lib/audio/default.sf2
 - jre/lib/security/cacerts

These should be fixed incrementally.

Reported-by: Oded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
mem: fix allocation accounting

There was a small bug in the free memory tracking code that I've only hit
recently. I was wrong in assuming that in the first branch for huge page
allocation, where we erase the entire range, we should account for N bytes.
This assumption came from my - wrong - understanding that we would do that when
the range is exactly N bytes.

Looking at the code with fresh eyes, that is definitely not what happens. In my
previous stress test we were hitting the second branch all the time, so this
bug lived on.

Turns out that we will delete the entire page range, which may be bigger than
N, the allocation size. Therefore, the whole range should be discounted from
our calculation. The remainder (bigger than N part) will be accounted for later
when we reinsert it in the page range, in the same way it is for the second
branch of this code.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
x64: Provide a backwards version of memcpy

This patch provides a backwards version of memcpy. It works all the same, but
will start the copy from dst + n <= src + n, instead of dst <= src. That is
needed for memmove when the source and destination operands overlap.

Being a nonstandard interface, I've just named it "memcpy_backwards"

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
string: implement memmove using memcpy

The current implementation of memmove is a PITA (I mean the bread, of course)
to decode if a fault happens. We have very little control of where exactly in
the code the fault happens, therefore it is difficult to reason about it. This
patch implements memmove in terms of memcpy + memcpy_backwards.

For those, we can have specific fixups in the possible fault sites, that will
allow us to decode the faults with ease.

Note that originally, the only reason why the first branch was not a memcpy is
that we would like to handle alignment. Since our implementation of memcpy is
fast enough, we can just ignore that and we will end up being even faster.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Update mgmt.git version

Changes:

  - web: Added /upload view class
  - Shell: Rewrite 'ls' and add formatting/sort flags
  - Update the jvm API to be more verbose
  - adding REST API specification: api, os, jvm

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
conf: make release build use -O3

In some workloads it noticably improves performance. I measured
6% increase in netperf throughput on my laptop.

Object file size is only slightly bloated:

 loader.elf (O2): 47246227
 loader.elf (O3): 51272625 (+8.5%)

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
module.py: move test.manifest.gen processing out of jvm class

Processing of this manifest was inside JVM-specific code which caused
the manifest was not processed if there was no java application
in the image.

For example:

  make image=empty check
  ...
  run_main(): cannot execute tests/tst-af-local.so. Powering off.
  Test tst-af-local.so FAILED
  make: *** [check] Error 1

Let's move it to the main manifest processing function.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
zfs: Fix on-disk data inconsistency on shutdown

This problem was found when running 'tests/tst-zfs-mount.so' multiple times.
At the first time, all tests succeed, however, a subsequent run would
fail at the test: 'mkdir /foo/bar', the error message reported
that the target file already exists.

The test basically creates a directory /foo/bar, rename it to /foo/bar2,
then remove /foo/bar2. How could /foo/bar still be there?

Quite simple. Our shutdown function calls unmount_rootfs() which will
attempt to unmount zfs with the flag MNT_FOURCE, however, it's not being
passed to zfs_unmount(), neither unmount_rootfs() tests itself the
return status (which was always getting failures previously).
So OSv is really being shutdown while there is remaining data waiting to
be synced with the backing store. As a result, inconsitency.

This problem was fixed by passing the flag to VFS_UNMOUNT which will now
unmount the fs properly on sudden shutdowns.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Add netperf image configuration

To start netserver inside OSv just do:

  make image=netperf
  sudo scripts/run.py -nv

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>