config.json: convert tabs to spaces

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: support nested module dependencies

Currently importing module from a module definition would fail because
we cannot call import module with the same name (module.py)
recursively, __import__ will complain that we removed 'module' from
sys.modules.

There is a simple solution to this problem, we can use runpy.run_path()
which works like a charm.

In addition to this we cache loaded modules so that we don't
have to load the file twice.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: make presence of either manifests optional

No need to create an empty bootfs.manifest anymore.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: cleanup resolve.py

No functional changes, just renames to more adequate names.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: make java a separate OSv module

This patch makes java files are copied to the guest image only when
'java' modue is included.

Modules can pull it explicitly by stating:

  require('java')

or implicitly, by creating api.run_java() run configurations.
In future we could consider moving api.run_java() into a java
meta-module.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: introduce higer level API to aid manifest generation

When plain manifests are not enough this is a concise alternative
with improved expresiveness. It allows to declare exclude and include
patterns. It's python based.

Example:

  m = FileMap()
  m.add('${OSV_BUILD_PATH}/tests').to('/tests') \
     .include('**/*.so') \
     .exclude('host/**')

Declared mappings can be saved in manifest form or be subject
of further processing. To save in manifest format:

  save_as_manifest(m, 'my.manifest')

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules: allow for file mapping declaration in module.py

If module has 'usr_files' or 'bootfs_files' declared then
their value will be interpreted as FileMaps and appended
to appropriate manifests.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
java: add all JRE files to the image by default

Fixes issue with JVM failing when started with a debugger
with the following message:

  NPT ERROR: Cannot find nptInitialize

Missing openjdk files in usr.manifest were a fertile source
of issues. This patch aims at making them less likely and
adding all files   except blacklisted files to the image.

This patch skips two files from JRE which are broken links
and inclusion of which would cause manifest upload failure:
 - jre/lib/audio/default.sf2
 - jre/lib/security/cacerts

These should be fixed incrementally.

Reported-by: Oded Gabbay <oded.gabbay@gmail.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
mem: fix allocation accounting

There was a small bug in the free memory tracking code that I've only hit
recently. I was wrong in assuming that in the first branch for huge page
allocation, where we erase the entire range, we should account for N bytes.
This assumption came from my - wrong - understanding that we would do that when
the range is exactly N bytes.

Looking at the code with fresh eyes, that is definitely not what happens. In my
previous stress test we were hitting the second branch all the time, so this
bug lived on.

Turns out that we will delete the entire page range, which may be bigger than
N, the allocation size. Therefore, the whole range should be discounted from
our calculation. The remainder (bigger than N part) will be accounted for later
when we reinsert it in the page range, in the same way it is for the second
branch of this code.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
x64: Provide a backwards version of memcpy

This patch provides a backwards version of memcpy. It works all the same, but
will start the copy from dst + n <= src + n, instead of dst <= src. That is
needed for memmove when the source and destination operands overlap.

Being a nonstandard interface, I've just named it "memcpy_backwards"

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
string: implement memmove using memcpy

The current implementation of memmove is a PITA (I mean the bread, of course)
to decode if a fault happens. We have very little control of where exactly in
the code the fault happens, therefore it is difficult to reason about it. This
patch implements memmove in terms of memcpy + memcpy_backwards.

For those, we can have specific fixups in the possible fault sites, that will
allow us to decode the faults with ease.

Note that originally, the only reason why the first branch was not a memcpy is
that we would like to handle alignment. Since our implementation of memcpy is
fast enough, we can just ignore that and we will end up being even faster.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Update mgmt.git version

Changes:

  - web: Added /upload view class
  - Shell: Rewrite 'ls' and add formatting/sort flags
  - Update the jvm API to be more verbose
  - adding REST API specification: api, os, jvm

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
conf: make release build use -O3

In some workloads it noticably improves performance. I measured
6% increase in netperf throughput on my laptop.

Object file size is only slightly bloated:

 loader.elf (O2): 47246227
 loader.elf (O3): 51272625 (+8.5%)

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
module.py: move test.manifest.gen processing out of jvm class

Processing of this manifest was inside JVM-specific code which caused
the manifest was not processed if there was no java application
in the image.

For example:

  make image=empty check
  ...
  run_main(): cannot execute tests/tst-af-local.so. Powering off.
  Test tst-af-local.so FAILED
  make: *** [check] Error 1

Let's move it to the main manifest processing function.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
zfs: Fix on-disk data inconsistency on shutdown

This problem was found when running 'tests/tst-zfs-mount.so' multiple times.
At the first time, all tests succeed, however, a subsequent run would
fail at the test: 'mkdir /foo/bar', the error message reported
that the target file already exists.

The test basically creates a directory /foo/bar, rename it to /foo/bar2,
then remove /foo/bar2. How could /foo/bar still be there?

Quite simple. Our shutdown function calls unmount_rootfs() which will
attempt to unmount zfs with the flag MNT_FOURCE, however, it's not being
passed to zfs_unmount(), neither unmount_rootfs() tests itself the
return status (which was always getting failures previously).
So OSv is really being shutdown while there is remaining data waiting to
be synced with the backing store. As a result, inconsitency.

This problem was fixed by passing the flag to VFS_UNMOUNT which will now
unmount the fs properly on sudden shutdowns.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Add netperf image configuration

To start netserver inside OSv just do:

  make image=netperf
  sudo scripts/run.py -nv

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
build: incremental make without image= argument should use the default

Currently the parameter was read from the generated Makefile which was
not re-generated on incremental build. The fix is to move the default
to build.mk, this way the default will always be picked unless masked
by command line argument.

Fixes #153

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
clock: add monotonic uptime clock

This patch starts to solve both issue #142 ("Support MONOTONIC_CLOCK")
and issue #81 (use <chrono> for time).

First, it adds an uptime() function to the "clock" interface, and
implements it for kvm/xen/hpet by returning the system time from which
we subtract the system time at boot (but not adding any correction
for wallclock).

Second, it adds a new std::chrono-based interface to this clock, in
a new header file <osv/clock.hh>. Instead of the old-style
clock::get()->uptime(), one should prefer osv::clock::uptime::now().
This returns a std::chrono::time_point which is type-safe, in the
sense that: 1. It knows what its epoch is (i.e., that it belongs to
osv::clock::uptime), and 2. It knows what its units are (nanoseconds).
This allows the compiler to prevent a user from confusing measurements
from this clock with those from other clocks, or making mistakes in
its units.

Third, this patch implements clock_gettime(MONOTONIC_CLOCK), using
the new osv::clock::uptime::now().

Note that though the new osv::clock::uptime is almost identical to
std::chrono::steady_clock, they should not be confused. The former is
actually OSv's implementation of the latter: steady_clock is implemented
by the C++11 standard library using the Posix clock_gettime, and that
is implemented (in this patch) using osv::clock::uptime.

With this patch, we're *not* done with either issues #142 or #81.
For issue #142, i.e., for supporting MONOTONIC_CLOCK in timerfd, we
need OSv's timers to work on uptime(), not on clock::get()->time().
For issue #81, we should add a osv::clock::wall type too (similar to
what clock::get()->time() does today, but more correctly), and use either
osv::clock::wall or osv::clock::uptime everywhere that
clock::get()->time() is currently used in the code.
clock::get()->time() should be removed.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
pci: Fix offsets in *_pci_config_*

On VMware, pci_readw(PCI_CFG_DEVICE_ID) returns the *vendor ID*.
pci_readw(PCI_CFG_VENDOR_ID) returns vendor ID as well.

Compare to FreeBSD implementation of read/write PCI config space,
FreeBSD masks lower bit of offset when write to PCI_CONFIG_ADDRESS, and
adds lower bit of offset to PCI_CONFIG_DATA.

http://fxr.watson.org/fxr/source/amd64/pci/pci_cfgreg.c#L206

This patch changes accessing method in OSv to the FreeBSD way.  Tested
on QEMU/KVM and VMware.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
string: add fixups for memcpy operations

When we start using the JVM balloon, our memcpy could fail for valid reasons
when the JVM is moving memory that is now in an unmapped region. To handle that,
register a fixup that will trigger a JVM call when the fault happens. If all goes
well, we will be able to continue normally.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>