top.py: also print total idle time

on a 4-cpu guest, top.py currently prints something like this:

	idle:  94%  75%  92%  92%

Where the first CPU is 94% idle, and the second CPU is 75% idle,
and so on. This patch adds another statistic, the sum of the CPUs'
idle times:

	idle:  94%  75%  92%  92%  (total 353%)

The number "353%" means that 353% of the strength of one CPU are idle.
This is often a useful statistic - on a fully loaded guest, we want to
see this get near to 0%.

The reason why we print the total idle percentage (353%) and not the
average (353/4 = 88%) is to keep all percentages in top.py consistent
in that they are of a single CPU, not of the total CPUs. If we start
printing both types of percentages on the same screen, it will be too
confusing.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
release-ec2: add --image-size option

Add --image-size option its suggest fs_size_mb when build image

Reviewed-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: KANATSU Minoru <icc.pot.tyew272@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
x64/misc.bin: openssl libraries

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Http Server: Add optional configuration file

This adds an optional configuration file under /etc/httpserver.conf

The configuration file is a Yaml formated file, with the optional server
configuration. It is been read before the server starts.

Configuration priority is now as follow, first configuration file,
second command line parameters are taken (though it is not very common
in httpserver) and last the default values.

This patch removes the default values from main, and leave it to be added
last in run method.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Cloud init: Add httpserver-module

httperver module allows setting the httpserver configuration by the cloud
init.

For example to set the port to 4000 and allow cross site access

httpserver:
  access-allow: true
  port: 4000

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Cloud-init: Change from a run command to a model

Include functionality should be identical regardless of the data source.
To make it clearer, that include is not only of script commands. The
include functionality was moved to its own module.

This patch also moves the load-from-cloud functionality to the include.

Now the local init file can have an option to read from cloud source.
This way it is possible to set an image with local configuration but
without attempting to call cloud source.

Cloud source has an option to decide how to handle missing cloud source.

This functionality was dropped when the cloud source were introduce and
is now put back.

The following options can be used with the include module:

include:
    - load-from-cloud:
        ignore-missing-source: true
    - file: /usr/mgmt/tmp.yaml
    - remote:
        server: 254.1.1.1
        path: /usr/data

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
httpserver: Export thread CPU via the API

Export the CPU which the thread is running on and display that in top.py.

Example top.py output looks like this:

301 threads on 4 CPUs; idle:  99%  99% 100% 100%  (total 398%)

   ID CPU  %CPU    TIME NAME
    2   1   1.1    0.74 page-access-sca
  264   0   0.1    0.02 >>>>>java.so
   45   0   0.1    0.03 rand_harvestq
  266   0   0.1    0.03 >>>>>java.so
  411   0   0.1    0.02 >>>>>java.so
  269   0   0.1    0.02 >>>>>java.so
  239   1   0.1    0.01 /libhttpserver.
   42   0   0.1    0.01 virtio-rng
  405   0   0.1    0.00 >>>>>java.so
  408   0   0.1    0.00 >>>>>java.so
  409   0   0.1    0.00 >>>>>java.so
  253   0   0.0    0.43 >>java.so
  254   1   0.0    0.35 >>java.so
  242   1   0.0    0.26 >java.so
  394   1   0.0    0.13 >>>>>java.so
  271   0   0.0    0.11 >>>>>java.so
  272   1   0.0    0.05 >>>>>java.so
  277   0   0.0    0.04 >>>>>java.so
  275   0   0.0    0.02 >>>>>java.so
   38   1   0.0    0.01 >init

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
top.py: Fix typo

Fix a typo in top.py that slipped in with commit 0911e53 ("httpserver: Export
thread CPU via the API").

Spotted by Nadav.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
elf: Fix SIGSEGV if dynamic table section is missing

If an ELF object does not have a dynamic table section, throw a proper
exception from _dynamic_tag() instead of faulting on nullptr.

Spotted while trying to run a static ELF binary.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc: add special file class to implement *mnt* functions

Few well known files like /etc/mtab, /etc/mnttab, or /proc/mounts gives
us view of mounted file systems. The APIs like getmntent, setmntent
etc are simple interfaces to work with these files. However, OSv does
not manage any of these well know files. Therefore, in order for these
functions to work, the mounted file system information is generated
from memory.

This patch implements a new class, call mtab_file, inherited from
special_file class. It allows us to implement all the standard file
operations.

This patch is essential for ZFS mount, unmount, and export to work
correctly. ZFS has its own mechanism to mount or unmount a ZFS file
system. For example: zfs mount command mounts a ZFS file system and
zfs unmount command unmounts a ZFS file system. zfs export command,
exports an earlier imported pool. When a pool is exported all the file
systems created and mounted must be unmounted before a pool can be
exported. Therefore, ZFS needs to have an exact view of currently
mounted file systems. ZFS uses getmntent function to read currently
mouted file systems.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
procfs: add mounts file

File /proc/mounts - maintains a list of all the filesystems currently
mounted. The format of data is similar to standard fstab format.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
zfs: fix umount and export functionality

zfs mount/umount - allows user to mount/unmount the given dataset.
zpool export - allows user to export already imported ZFS pool. The
export functionality has to ensure that no dataset (file system)
belonging to exported pool is mounted.

Both zfs unmount/mount/export rely on common functionality implemented
in libzfs to give correct list of mounted file systems. libzfs reads
the mount information from /etc/mnttab. With this patch /etc/mnttab is
a symlink to /proc/mounts.

ZFS code uses BSD specific mechanism to find list of mount entries.
Specifically, BSD uses different structure called mnttab as opposed to
mntent used on Linux. The API getmntent() on BSD and Linux assumes
different prototype. This patch introduces new BSD porting functions
to map to Linux equivalent.

Reviewed-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules/lua: fix build failure

Building cli image fails with following error -

$ make -j 6 mode=release image=cli

make[2]: Entering directory `.../osv/modules/lua'
cd src && make
cd build/luarocks-2.1.2 && \
    ./configure --prefix=.../osv/modules/lua/out --force-config \
    --with-lua-include=.../osv/modules/lua/src --lua-version=5.2
make[3]: Entering directory `/.../osv/modules/lua/src'
gcc -O2 -Wall -fPIC -DLUA_COMPAT_ALL -DLUA_USE_LINUX -shared -o \
   liblua.so  lapi.o ...........
gcc -o lua  lua.o liblua.a -lm -ldl -ledit
make[3]: Leaving directory `.../osv/modules/lua/src'
cp src/lua out/bin/lua
lua not found in $PATH.
You may want to use the flags --with-lua and/or --lua-suffix.
See --help.

configure failed.

make[2]: *** [build/luarocks-2.1.2/Makefile] Error 1
make[2]: Leaving directory `.../osv/modules/lua'

The patch fixes build for me.

Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Http Server: Add test for thread CPU

To test that the cpu parameter correct, the test check that the cpu of
idle1 (the idle thread of cpu1) is equal to 1.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc: add mount(2)

Linux definition of mount requires the parameter data to be const,
so changes were made whenever needed to follow that.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
pthread: call dtors for pthread thread specific data

Use register_exit_notifier() to call dtors of thread specific data.

Right now, C++'s "thread_local" variable deconstruction relies on
pthread's task specific data management functions to work. So this
patch also fixes #373.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
tests: add tests for pthread tsd related functions

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
sched: remove abandon_current() call in thread::complete()

The application::abandon_current() call used to workaround the
thread_local deconstruction issue by explicitly deconstructing the current_app
variable.

This is no longer needed since thread_local deconstruction now works with the
previous patch. The variable will be deconstructed automatically when thread
exits.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
sched: adopt application in thread context

application::adopt() used to access other thread's thread_local variable
via remote_thread_local_var(). However c++'s thread_local variables are
initialized on first use, and compilers are not clever enough to
recognize
the indirect "use" through remote_thread_local_val(). So, the variable
is
not properly initialized before we access it.

Particularly, the deconstructor for 'current_app' is not registered to
be called when thread exits which shall be done during initialization.
This will cause the use_count of 'current_app' not decreasing after
thread
exits. As a result, the application pointed by 'current_app' is not
deleted.

This patch moves the call to adopt_current() in the thread context, so that
the thread_local are properly initialized. The 'app' to adopt is passed
via a lambda expression wrapping '_func'.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
app: remove application::adopt()

This function is no longer needed. Use adopt_current() in thread context
instead.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>