httpserver: Export thread CPU via the API

Export the CPU which the thread is running on and display that in top.py.

Example top.py output looks like this:

301 threads on 4 CPUs; idle:  99%  99% 100% 100%  (total 398%)

   ID CPU  %CPU    TIME NAME
    2   1   1.1    0.74 page-access-sca
  264   0   0.1    0.02 >>>>>java.so
   45   0   0.1    0.03 rand_harvestq
  266   0   0.1    0.03 >>>>>java.so
  411   0   0.1    0.02 >>>>>java.so
  269   0   0.1    0.02 >>>>>java.so
  239   1   0.1    0.01 /libhttpserver.
   42   0   0.1    0.01 virtio-rng
  405   0   0.1    0.00 >>>>>java.so
  408   0   0.1    0.00 >>>>>java.so
  409   0   0.1    0.00 >>>>>java.so
  253   0   0.0    0.43 >>java.so
  254   1   0.0    0.35 >>java.so
  242   1   0.0    0.26 >java.so
  394   1   0.0    0.13 >>>>>java.so
  271   0   0.0    0.11 >>>>>java.so
  272   1   0.0    0.05 >>>>>java.so
  277   0   0.0    0.04 >>>>>java.so
  275   0   0.0    0.02 >>>>>java.so
   38   1   0.0    0.01 >init

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
top.py: Fix typo

Fix a typo in top.py that slipped in with commit 0911e53 ("httpserver: Export
thread CPU via the API").

Spotted by Nadav.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
elf: Fix SIGSEGV if dynamic table section is missing

If an ELF object does not have a dynamic table section, throw a proper
exception from _dynamic_tag() instead of faulting on nullptr.

Spotted while trying to run a static ELF binary.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc: add special file class to implement *mnt* functions

Few well known files like /etc/mtab, /etc/mnttab, or /proc/mounts gives
us view of mounted file systems. The APIs like getmntent, setmntent
etc are simple interfaces to work with these files. However, OSv does
not manage any of these well know files. Therefore, in order for these
functions to work, the mounted file system information is generated
from memory.

This patch implements a new class, call mtab_file, inherited from
special_file class. It allows us to implement all the standard file
operations.

This patch is essential for ZFS mount, unmount, and export to work
correctly. ZFS has its own mechanism to mount or unmount a ZFS file
system. For example: zfs mount command mounts a ZFS file system and
zfs unmount command unmounts a ZFS file system. zfs export command,
exports an earlier imported pool. When a pool is exported all the file
systems created and mounted must be unmounted before a pool can be
exported. Therefore, ZFS needs to have an exact view of currently
mounted file systems. ZFS uses getmntent function to read currently
mouted file systems.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
procfs: add mounts file

File /proc/mounts - maintains a list of all the filesystems currently
mounted. The format of data is similar to standard fstab format.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
zfs: fix umount and export functionality

zfs mount/umount - allows user to mount/unmount the given dataset.
zpool export - allows user to export already imported ZFS pool. The
export functionality has to ensure that no dataset (file system)
belonging to exported pool is mounted.

Both zfs unmount/mount/export rely on common functionality implemented
in libzfs to give correct list of mounted file systems. libzfs reads
the mount information from /etc/mnttab. With this patch /etc/mnttab is
a symlink to /proc/mounts.

ZFS code uses BSD specific mechanism to find list of mount entries.
Specifically, BSD uses different structure called mnttab as opposed to
mntent used on Linux. The API getmntent() on BSD and Linux assumes
different prototype. This patch introduces new BSD porting functions
to map to Linux equivalent.

Reviewed-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
modules/lua: fix build failure

Building cli image fails with following error -

$ make -j 6 mode=release image=cli

make[2]: Entering directory `.../osv/modules/lua'
cd src && make
cd build/luarocks-2.1.2 && \
    ./configure --prefix=.../osv/modules/lua/out --force-config \
    --with-lua-include=.../osv/modules/lua/src --lua-version=5.2
make[3]: Entering directory `/.../osv/modules/lua/src'
gcc -O2 -Wall -fPIC -DLUA_COMPAT_ALL -DLUA_USE_LINUX -shared -o \
   liblua.so  lapi.o ...........
gcc -o lua  lua.o liblua.a -lm -ldl -ledit
make[3]: Leaving directory `.../osv/modules/lua/src'
cp src/lua out/bin/lua
lua not found in $PATH.
You may want to use the flags --with-lua and/or --lua-suffix.
See --help.

configure failed.

make[2]: *** [build/luarocks-2.1.2/Makefile] Error 1
make[2]: Leaving directory `.../osv/modules/lua'

The patch fixes build for me.

Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Http Server: Add test for thread CPU

To test that the cpu parameter correct, the test check that the cpu of
idle1 (the idle thread of cpu1) is equal to 1.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc: add mount(2)

Linux definition of mount requires the parameter data to be const,
so changes were made whenever needed to follow that.

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
pthread: call dtors for pthread thread specific data

Use register_exit_notifier() to call dtors of thread specific data.

Right now, C++'s "thread_local" variable deconstruction relies on
pthread's task specific data management functions to work. So this
patch also fixes #373.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
tests: add tests for pthread tsd related functions

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
sched: remove abandon_current() call in thread::complete()

The application::abandon_current() call used to workaround the
thread_local deconstruction issue by explicitly deconstructing the current_app
variable.

This is no longer needed since thread_local deconstruction now works with the
previous patch. The variable will be deconstructed automatically when thread
exits.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
sched: adopt application in thread context

application::adopt() used to access other thread's thread_local variable
via remote_thread_local_var(). However c++'s thread_local variables are
initialized on first use, and compilers are not clever enough to
recognize
the indirect "use" through remote_thread_local_val(). So, the variable
is
not properly initialized before we access it.

Particularly, the deconstructor for 'current_app' is not registered to
be called when thread exits which shall be done during initialization.
This will cause the use_count of 'current_app' not decreasing after
thread
exits. As a result, the application pointed by 'current_app' is not
deleted.

This patch moves the call to adopt_current() in the thread context, so that
the thread_local are properly initialized. The 'app' to adopt is passed
via a lambda expression wrapping '_func'.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
app: remove application::adopt()

This function is no longer needed. Use adopt_current() in thread context
instead.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Http Server: Returned error messages should be json objects

When using exception to indicate an error message, the return body
should be a json object.

This patch uses a json helper class to translate an exception to a
formatted json object.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Remove obsolete TODO.zfs file

The file has not been updated since May 2013 so perhaps it's time to finally
remove it...

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
sched: add "sched_preempt" tracepoint

Before commit 5f5b6144a5950e5b6897450a51f424ab28df3812, we had a
sched_preempt tracepoint. The tracepoint still exists, but it wasn't
used. This patch uses it again, with a different meaning.

What "sched_preempt" marks now is a case where a running thread
was preempted. I.e., the scheduler switched to a different thread
despite the original thread still being runnable.

Note that every sched_preempt event is also a sched_switch event.

In practice, a "sched_preempt" can happen in one of four basic cases:

1. The running thread wake()s another thread on the same CPU, and
   the scheduler decides to switch to the woken thread instead of
   continuing to run the waker.

2. Similarly, an interrupt handler can wake() a thread on this CPU
   and reschedule, and if the scheduler decides to switch threads,
   this is a preemption. This happens, for example, when a thread
   waits on a timer.

3. The preemption timer expired, causing the scheduler to run and
   notice that another thread has become the one with the lowest
   runtime, so it runs that thread instead of the running one.

4. Another CPU woke up one of this CPU's threads, and sent this CPU
   an IPI. The sheduler runs and can decide to switch to the newly
   woken thread.

Note: currently, when a 1-vcpu OSv is completely idle, we see 1000
sched_preempt's per second. This is expected, because of the
page_scanner_thread: This thread timer to wake it 1000 times a second,
and each time it is woken it preempts the *idle thread* which is
runnable all the time.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
gdb: rework 'osv zfs' to provide better ARC data

New output:
(gdb) osv zfs
:: ZFS TUNABLES ::
	zil_replay_disable:       0
	zfs_nocacheflush:         0
	zfs_prefetch_disable:     0
	zfs_no_write_throttle:    0
	zfs_txg_timeout:          5
	zfs_write_limit_override: 0
	vdev_min_pending:         4
	vdev_max_pending:         10

:: ARC SIZES ::
	Current size:    1493512 (1 MB)
	Target size:     1341923840 (1279 MB)
	Min target size: 167740480 (159 MB)
	Max target size: 1341923840 (1279 MB)

:: ARC SIZE BREAKDOWN ::
	Most recently used cache size:   670961920 (639 MB) (50.00%)
	Most frequently used cache size: 670961920 (639 MB) (50.00%)

:: ARC EFFICIENCY ::
	Cache access total:  136422
	Cache hit ratio:     132326 (97.00%)
	Cache miss ratio:    4096 (3.00%)
	REAL hit ratio:      132751 (97.31%)

	Data demand efficiency:   99.99%
	Data prefetch efficiency: 0.00%

:: CACHE HITS BY CACHE LIST ::
	Most recently used:           108539 (82.02%)
	Most frequently used:         24212 (18.30%)
	Most recently used (Ghost):   0 (0.00%)
	Most frequently used (Ghost): 0 (0.00%)

:: CACHE HITS BY DATA TYPE ::
	Demand data:       127471 (96.33%)
	Prefetch data:     0 (0.00%)
	Demand metadata:   4855 (3.67%)
	Prefetch metadata: 0 (0.00%)

:: CACHE MISSES BY DATA TYPE ::
	Demand data:       8 (0.20%)
	Prefetch data:     3998 (97.61%)
	Demand metadata:   86 (2.10%)
	Prefetch metadata: 4 (0.10%)

:: ARC HASH INFO ::
	Total hash elements: 4103
	Max hash elements:   4103
	Hash collisions:     77
	Hash chains:         55

Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
trace: show thread name in 'trace.py prof'

Applies when grouping by thread.

Before:

  === Thread 0xffff8001170c3050 ===

After:

  === Thread >>>>>>>java.so (0xffff8001170c3050) ===

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
zfs: add missing ZFS_EXIT()

Not that it fixes something since ZFS_EXIT is currently no-op on OSv
but if one day we would like to bring it back this would matter.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>