libc/misc: Migrate to upstream musl

Migrate all identical files under libc/misc to upstream musl.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc/passwd: Migrate to upstream musl

Migrate all identical files under libc/passwd to upstream musl.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc/thread: Migrate to upstream musl

Migrate all identical files under libc/thread to upstream musl.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc: Move _chk variants into separate files

Move the _chk variants into separate files so to simplify migration to
upstream musl.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
trace.py: optimize split_format()

This improves all trace.py commands by 30%. We're using memoization
techinque to save computation.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
trace.py: optimize unpack()

This brings a small but noticable improvement.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
httpserver: fix JSON boolean formatting

The JSON standard specifies that boolean values should be written
as 'true' and 'false' (without the quotes). The standard goes as
far as specifying the ASCII values of these letters, and in particular
it is case sensitive - one cannot write True or TRUE as a synonym of true.

We erronenously used True/False with a capital letters. Python's
JSON parser, for example, fails to recognize such broken JSON.
So this patch fixes this.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Revert "trace.py: optimize unpack()"

This reverts commit c077d1fd8efbc007e137262962c8a504bd269eee.

The change breaks 'trace extract':

[penberg@localhost osv]$ trace extract
warning: Unable to find dynamic linker breakpoint function.
GDB will be unable to debug shared library initializers
and track explicitly loaded dynamic code.
sched::cpu::do_idle (this=this@entry=0xffff80000113a040) at
/home/penberg/osv/core/sched.cc:388
388             handle_incoming_wakeups();
Python Exception <type 'exceptions.TypeError'> unpack_from() argument
1 must be convertible to a buffer, not concat:
Saving traces to tracefile ...
Error occurred in Python command: unpack_from() argument 1 must be
convertible to a buffer, not concat

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
cloud-init: fix spelling mistake in error message

Observed this becuase cloud-init fails on my setup.

Signed-off-by: Prasad Joshi <prasadjoshi.linux@gmail.com>
Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
vfs: stop polls before calling close()

After close() is called, the file should not be used any more. We need
to stop pollers before doing that, otherwise epoll_wait() can attempt
calling poll() on a file which was closed. Doing so will result in
aborts like the one below, because the socket, which is attempted to
be locked has been already freed.

[backtrace]
0x00000000002232a8 <__assert_fail+24>
0x00000000003a7e27 <lockfree::mutex::unlock()+487>
0x00000000003dfe0d <epoll_file::wait(epoll_event*, int, int)+1165>
0x00000000003de240 <epoll_wait+112>
0x0000100004409ad2 <Java_sun_nio_ch_EPollArrayWrapper_epollWait+210>

Fixes #424.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
tracepoints: make tracepoint_counter generally useful

tracepoint_counter is a useful class which allows counting the number
of times that a particular tracepoint occured, *without* enabling
the tracepoint and post-processing the tracepoint log.

tracepoint_counter works by setting up the tracepoint to, every time it
is reached, increment a per-cpu counter (without logging the tracepoint).
This process is very efficient, and incurs only a small performance
overhead - even for tracepoints reached millions of times a second
(where logging the events and post-processing them would be totally
unrealistic).

Previously, tracepoint_counter lived in the tracepoint.cc, a JNI library
which we used to implement the "perf stat" command in the Java-based
shell. This patch extracts this class to a new header file,
<osv/trace-count.hh>, where we can use it not just for Java code.
In the next patch, we intend to use the same class to provide
tracepoint counting also for the REST API.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
httpserver: add operations for counting tracepoints

This patch adds to httpserver's REST API the ability to efficiently count
the number of times that a particular tracepoint is reached, by using the
tracepoint_counter OSv feature provided by the previous patch.
In the next patch we will use this REST API to provide a "freq.py"
script which can efficiently display the frequency of various tracepoints,
from outside the guest.

This patch adds three operations:

/trace/count/<tracepoint-name> (POST): enable or disable the counting of
a particular tracepoint. When counting of a tracepoint is enabled, each
time this tracepoint is reached a counter is efficiently (per-cpu)
incremented. This counting continues indefinitely until the counting of
this tracepoint is later disabled. Disabling and later renabling a
counter starts again with the zero count. Similarly, enabling an already-
enabled counter also zeros the counter.

/trace/count (DELETE): a shortcut for disabling all enabled counter.

/trace/count (GET): retrieve the counts for all enabled tracepoint counters.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
add "freq.py" tool to show tracepoint frequency in a running OSv guest

This patch adds a tool "freq.py" which can be used from outside the OSv
guest to find the frequency of *any* tracepoint. This is done efficiently
(see tracepoint_counter) and does not involve fully enabling (logging)
this tracepoint. It can even be used for tracepoints which occur
millions a time each second.

Freq.py takes the guest's address:port (the port defaults to 8000), and
a list of tracepoints whose frequency to display. To just see the whole
list of tracepoints to choose from, run freq.py Without parameters
(except the host). The frequency of each selected tracepoint (number of
occurances per second) is averaged over 2-second periods and displayed
every 2 seconds.

Here is an example of the output for three tracepoints on a guest
running memcached, before and after "memaslap" started exercising it:

$ scripts/freq.py 192.168.122.89 sched_switch mutex_lock memory_malloc
Showing number of events per second. Refreshing every 2 seconds
   sched_switch     mutex_lock  memory_malloc
           2650           4057            265
           2274           1298            163
           2033           1665            175
           3447           6344            343
          52782         805701         227800  <--- memaslap started here
          78631        1068659         304723
          84144        1060092         301155

To reiterate, you *do not* need to enable tracepoints in the traditional
sense (e.g., the "--trace ..." paramters to run.py) for this to work,
and you can ask freq.py to count *any* existing tracepoint.
The counted tracepoints are only counted - not logged to the trace log
(of course, you can do both if you wish).

One caveat on this implementation:

By design, the REST API is session-less. When freq.py enables a counter,
the kernel will never stop counting it - until freq.py asks it to stop
counting. We were careful about stopping all counters when freq.py exits
(even if it is killed by a signal), but this can still happen - e.g., if
the host running freq.py dies. Moreover, freq.py is currently not designed
to support two freq.py runs on the same OSv guest concurrently. As the
code is written now, the second concurrent freq.py invocation will cancel
all the counters requested by the first invocation, and confuse it.
This situation can be fixed, but not easily (will involve some sort of
reference counting?) but I'll leave it to a follow-up patch. I think
this patch is already quite useful as-is.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
runtime: make sure halt() is not called before abort() message is printed

Concurrent abort() call will call osv::halt() immediately which may
result in a halting NMI being delivered to the CPU which is in the
middle of printing the abort message. This can lead to silent aborts
with no message on the screen. To make debugging easier, let the first
aborter call osv::halt(), others will just spin and wait for the
leader to halt all CPUs.

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
top.py: also print total idle time

on a 4-cpu guest, top.py currently prints something like this:

	idle:  94%  75%  92%  92%

Where the first CPU is 94% idle, and the second CPU is 75% idle,
and so on. This patch adds another statistic, the sum of the CPUs'
idle times:

	idle:  94%  75%  92%  92%  (total 353%)

The number "353%" means that 353% of the strength of one CPU are idle.
This is often a useful statistic - on a fully loaded guest, we want to
see this get near to 0%.

The reason why we print the total idle percentage (353%) and not the
average (353/4 = 88%) is to keep all percentages in top.py consistent
in that they are of a single CPU, not of the total CPUs. If we start
printing both types of percentages on the same screen, it will be too
confusing.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
release-ec2: add --image-size option

Add --image-size option its suggest fs_size_mb when build image

Reviewed-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: KANATSU Minoru <icc.pot.tyew272@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
x64/misc.bin: openssl libraries

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Http Server: Add optional configuration file

This adds an optional configuration file under /etc/httpserver.conf

The configuration file is a Yaml formated file, with the optional server
configuration. It is been read before the server starts.

Configuration priority is now as follow, first configuration file,
second command line parameters are taken (though it is not very common
in httpserver) and last the default values.

This patch removes the default values from main, and leave it to be added
last in run method.

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Cloud init: Add httpserver-module

httperver module allows setting the httpserver configuration by the cloud
init.

For example to set the port to 4000 and allow cross site access

httpserver:
  access-allow: true
  port: 4000

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Cloud-init: Change from a run command to a model

Include functionality should be identical regardless of the data source.
To make it clearer, that include is not only of script commands. The
include functionality was moved to its own module.

This patch also moves the load-from-cloud functionality to the include.

Now the local init file can have an option to read from cloud source.
This way it is possible to set an image with local configuration but
without attempting to call cloud source.

Cloud source has an option to decide how to handle missing cloud source.

This functionality was dropped when the cloud source were introduce and
is now put back.

The following options can be used with the include module:

include:
    - load-from-cloud:
        ignore-missing-source: true
    - file: /usr/mgmt/tmp.yaml
    - remote:
        server: 254.1.1.1
        path: /usr/data

Signed-off-by: Amnon Heiman <amnon@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>