vfs: stop polls before calling close()

After close() is called, the file should not be used any more. We need
to stop pollers before doing that, otherwise epoll_wait() can attempt
calling poll() on a file which was closed. Doing so will result in
aborts like the one below, because the socket, which is attempted to
be locked has been already freed.

[backtrace]
0x00000000002232a8 <__assert_fail+24>
0x00000000003a7e27 <lockfree::mutex::unlock()+487>
0x00000000003dfe0d <epoll_file::wait(epoll_event*, int, int)+1165>
0x00000000003de240 <epoll_wait+112>
0x0000100004409ad2 <Java_sun_nio_ch_EPollArrayWrapper_epollWait+210>

Fixes #424.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
tracepoints: make tracepoint_counter generally useful

tracepoint_counter is a useful class which allows counting the number
of times that a particular tracepoint occured, *without* enabling
the tracepoint and post-processing the tracepoint log.

tracepoint_counter works by setting up the tracepoint to, every time it
is reached, increment a per-cpu counter (without logging the tracepoint).
This process is very efficient, and incurs only a small performance
overhead - even for tracepoints reached millions of times a second
(where logging the events and post-processing them would be totally
unrealistic).

Previously, tracepoint_counter lived in the tracepoint.cc, a JNI library
which we used to implement the "perf stat" command in the Java-based
shell. This patch extracts this class to a new header file,
<osv/trace-count.hh>, where we can use it not just for Java code.
In the next patch, we intend to use the same class to provide
tracepoint counting also for the REST API.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
httpserver: add operations for counting tracepoints

This patch adds to httpserver's REST API the ability to efficiently count
the number of times that a particular tracepoint is reached, by using the
tracepoint_counter OSv feature provided by the previous patch.
In the next patch we will use this REST API to provide a "freq.py"
script which can efficiently display the frequency of various tracepoints,
from outside the guest.

This patch adds three operations:

/trace/count/<tracepoint-name> (POST): enable or disable the counting of
a particular tracepoint. When counting of a tracepoint is enabled, each
time this tracepoint is reached a counter is efficiently (per-cpu)
incremented. This counting continues indefinitely until the counting of
this tracepoint is later disabled. Disabling and later renabling a
counter starts again with the zero count. Similarly, enabling an already-
enabled counter also zeros the counter.

/trace/count (DELETE): a shortcut for disabling all enabled counter.

/trace/count (GET): retrieve the counts for all enabled tracepoint counters.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
add "freq.py" tool to show tracepoint frequency in a running OSv guest

This patch adds a tool "freq.py" which can be used from outside the OSv
guest to find the frequency of *any* tracepoint. This is done efficiently
(see tracepoint_counter) and does not involve fully enabling (logging)
this tracepoint. It can even be used for tracepoints which occur
millions a time each second.

Freq.py takes the guest's address:port (the port defaults to 8000), and
a list of tracepoints whose frequency to display. To just see the whole
list of tracepoints to choose from, run freq.py Without parameters
(except the host). The frequency of each selected tracepoint (number of
occurances per second) is averaged over 2-second periods and displayed
every 2 seconds.

Here is an example of the output for three tracepoints on a guest
running memcached, before and after "memaslap" started exercising it:

$ scripts/freq.py 192.168.122.89 sched_switch mutex_lock memory_malloc
Showing number of events per second. Refreshing every 2 seconds
   sched_switch     mutex_lock  memory_malloc
           2650           4057            265
           2274           1298            163
           2033           1665            175
           3447           6344            343
          52782         805701         227800  <--- memaslap started here
          78631        1068659         304723
          84144        1060092         301155

To reiterate, you *do not* need to enable tracepoints in the traditional
sense (e.g., the "--trace ..." paramters to run.py) for this to work,
and you can ask freq.py to count *any* existing tracepoint.
The counted tracepoints are only counted - not logged to the trace log
(of course, you can do both if you wish).

One caveat on this implementation:

By design, the REST API is session-less. When freq.py enables a counter,
the kernel will never stop counting it - until freq.py asks it to stop
counting. We were careful about stopping all counters when freq.py exits
(even if it is killed by a signal), but this can still happen - e.g., if
the host running freq.py dies. Moreover, freq.py is currently not designed
to support two freq.py runs on the same OSv guest concurrently. As the
code is written now, the second concurrent freq.py invocation will cancel
all the counters requested by the first invocation, and confuse it.
This situation can be fixed, but not easily (will involve some sort of
reference counting?) but I'll leave it to a follow-up patch. I think
this patch is already quite useful as-is.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
runtime: make sure halt() is not called before abort() message is printed

Concurrent abort() call will call osv::halt() immediately which may
result in a halting NMI being delivered to the CPU which is in the
middle of printing the abort message. This can lead to silent aborts
with no message on the screen. To make debugging easier, let the first
aborter call osv::halt(), others will just spin and wait for the
leader to halt all CPUs.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
top.py: also print total idle time

on a 4-cpu guest, top.py currently prints something like this:

	idle:  94%  75%  92%  92%

Where the first CPU is 94% idle, and the second CPU is 75% idle,
and so on. This patch adds another statistic, the sum of the CPUs'
idle times:

	idle:  94%  75%  92%  92%  (total 353%)

The number "353%" means that 353% of the strength of one CPU are idle.
This is often a useful statistic - on a fully loaded guest, we want to
see this get near to 0%.

The reason why we print the total idle percentage (353%) and not the
average (353/4 = 88%) is to keep all percentages in top.py consistent
in that they are of a single CPU, not of the total CPUs. If we start
printing both types of percentages on the same screen, it will be too
confusing.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
release-ec2: add --image-size option

Add --image-size option its suggest fs_size_mb when build image

Reviewed-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: KANATSU Minoru <icc.pot.tyew272@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
x64/misc.bin: openssl libraries

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Http Server: Add optional configuration file

This adds an optional configuration file under /etc/httpserver.conf

The configuration file is a Yaml formated file, with the optional server
configuration. It is been read before the server starts.

Configuration priority is now as follow, first configuration file,
second command line parameters are taken (though it is not very common
in httpserver) and last the default values.

This patch removes the default values from main, and leave it to be added
last in run method.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Cloud init: Add httpserver-module

httperver module allows setting the httpserver configuration by the cloud
init.

For example to set the port to 4000 and allow cross site access

httpserver:
  access-allow: true
  port: 4000

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Cloud-init: Change from a run command to a model

Include functionality should be identical regardless of the data source.
To make it clearer, that include is not only of script commands. The
include functionality was moved to its own module.

This patch also moves the load-from-cloud functionality to the include.

Now the local init file can have an option to read from cloud source.
This way it is possible to set an image with local configuration but
without attempting to call cloud source.

Cloud source has an option to decide how to handle missing cloud source.

This functionality was dropped when the cloud source were introduce and
is now put back.

The following options can be used with the include module:

include:
    - load-from-cloud:
        ignore-missing-source: true
    - file: /usr/mgmt/tmp.yaml
    - remote:
        server: 254.1.1.1
        path: /usr/data

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
httpserver: Export thread CPU via the API

Export the CPU which the thread is running on and display that in top.py.

Example top.py output looks like this:

301 threads on 4 CPUs; idle:  99%  99% 100% 100%  (total 398%)

   ID CPU  %CPU    TIME NAME
    2   1   1.1    0.74 page-access-sca
  264   0   0.1    0.02 >>>>>java.so
   45   0   0.1    0.03 rand_harvestq
  266   0   0.1    0.03 >>>>>java.so
  411   0   0.1    0.02 >>>>>java.so
  269   0   0.1    0.02 >>>>>java.so
  239   1   0.1    0.01 /libhttpserver.
   42   0   0.1    0.01 virtio-rng
  405   0   0.1    0.00 >>>>>java.so
  408   0   0.1    0.00 >>>>>java.so
  409   0   0.1    0.00 >>>>>java.so
  253   0   0.0    0.43 >>java.so
  254   1   0.0    0.35 >>java.so
  242   1   0.0    0.26 >java.so
  394   1   0.0    0.13 >>>>>java.so
  271   0   0.0    0.11 >>>>>java.so
  272   1   0.0    0.05 >>>>>java.so
  277   0   0.0    0.04 >>>>>java.so
  275   0   0.0    0.02 >>>>>java.so
   38   1   0.0    0.01 >init

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
top.py: Fix typo

Fix a typo in top.py that slipped in with commit 0911e53 ("httpserver: Export
thread CPU via the API").

Spotted by Nadav.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
elf: Fix SIGSEGV if dynamic table section is missing

If an ELF object does not have a dynamic table section, throw a proper
exception from _dynamic_tag() instead of faulting on nullptr.

Spotted while trying to run a static ELF binary.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc: add special file class to implement *mnt* functions

Few well known files like /etc/mtab, /etc/mnttab, or /proc/mounts gives
us view of mounted file systems. The APIs like getmntent, setmntent
etc are simple interfaces to work with these files. However, OSv does
not manage any of these well know files. Therefore, in order for these
functions to work, the mounted file system information is generated
from memory.

This patch implements a new class, call mtab_file, inherited from
special_file class. It allows us to implement all the standard file
operations.

This patch is essential for ZFS mount, unmount, and export to work
correctly. ZFS has its own mechanism to mount or unmount a ZFS file
system. For example: zfs mount command mounts a ZFS file system and
zfs unmount command unmounts a ZFS file system. zfs export command,
exports an earlier imported pool. When a pool is exported all the file
systems created and mounted must be unmounted before a pool can be
exported. Therefore, ZFS needs to have an exact view of currently
mounted file systems. ZFS uses getmntent function to read currently
mouted file systems.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
procfs: add mounts file

File /proc/mounts - maintains a list of all the filesystems currently
mounted. The format of data is similar to standard fstab format.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
zfs: fix umount and export functionality

zfs mount/umount - allows user to mount/unmount the given dataset.
zpool export - allows user to export already imported ZFS pool. The
export functionality has to ensure that no dataset (file system)
belonging to exported pool is mounted.

Both zfs unmount/mount/export rely on common functionality implemented
in libzfs to give correct list of mounted file systems. libzfs reads
the mount information from /etc/mnttab. With this patch /etc/mnttab is
a symlink to /proc/mounts.

ZFS code uses BSD specific mechanism to find list of mount entries.
Specifically, BSD uses different structure called mnttab as opposed to
mntent used on Linux. The API getmntent() on BSD and Linux assumes
different prototype. This patch introduces new BSD porting functions
to map to Linux equivalent.

Reviewed-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules/lua: fix build failure

Building cli image fails with following error -

$ make -j 6 mode=release image=cli

make[2]: Entering directory `.../osv/modules/lua'
cd src && make
cd build/luarocks-2.1.2 && \
    ./configure --prefix=.../osv/modules/lua/out --force-config \
    --with-lua-include=.../osv/modules/lua/src --lua-version=5.2
make[3]: Entering directory `/.../osv/modules/lua/src'
gcc -O2 -Wall -fPIC -DLUA_COMPAT_ALL -DLUA_USE_LINUX -shared -o \
   liblua.so  lapi.o ...........
gcc -o lua  lua.o liblua.a -lm -ldl -ledit
make[3]: Leaving directory `.../osv/modules/lua/src'
cp src/lua out/bin/lua
lua not found in $PATH.
You may want to use the flags --with-lua and/or --lua-suffix.
See --help.

configure failed.

make[2]: *** [build/luarocks-2.1.2/Makefile] Error 1
make[2]: Leaving directory `.../osv/modules/lua'

The patch fixes build for me.

Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Http Server: Add test for thread CPU

To test that the cpu parameter correct, the test check that the cpu of
idle1 (the idle thread of cpu1) is equal to 1.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc: add mount(2)

Linux definition of mount requires the parameter data to be const,
so changes were made whenever needed to follow that.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>