dhcp: remove lookup_opcode()

The lookup_opcode() function is incorrect. It was mishandling
DHCP_OPTION_PAD, which does not have a following length byte.

Also, the while condition is reading 'op' value which never
changes. This may result in reads beyond packet size.

Since this function is unused the best fix is to remove it.

Reveiwed-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
dhcp: fix parsing of DHCP options

There are several issues with current code. Firstly, the LENGTH_OK
macro was used in the condition in while(). This macro was checking if
options + op_len does not exceed packet limit. This macro works fine
when used from PARSE_OP() inside the switch, but because 'options' is
bumped up by op_len at the end of the loop body, use of this macro in
the while condition may result in premature exit of the loop. This was
causing that some times OSv was not parsing network mask and gateway
leaving them at 0.0.0.0 when started on Goodle Compute Engine. As a
result OSv was not responding over network. See issue #254.

Another issue was that the stop condition which checks for op ==
DHCP_OPTION_END was using 'op' from the outer context, which was never
overwritten. The actual variable which was changed based on the packet
content was redeclared inside the loop.

A third problem, spotted by Vlad, is that the code was not handling
DHCP_OPTION_PAD properly. This option has only opcode byte and no
following length byte. Currrent code would attempt to read the length
byte and skip by that amount, which would yiled incorrect parsing
result.

Reviewed-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
zfs: spa_zio_taskq[ZIO_TYPE_FREE][ZIO_TASKQ_ISSUE]->tq_lock contention

Reviewed by: Matthew Ahrens <mahrens@delphix.com>
Reviewed by: George Wilson <george.wilson@delphix.com>
Reviewed by: Christopher Siden <christopher.siden@delphix.com>
Reviewed by: Gordon Ross <gordon.ross@nexenta.com>
Approved by: Richard Lowe <richlowe@richlowe.net>

Reference: https://illumos.org/issues/3581

Patch taken from Illumos and slight changes were needed to port it to OSv.

This patch targets improvement on taskq lock contention by dispatching work
over independent task queues. ZFS on Linux devops mention that it's not clear
whether or not this issue affects their port, but profile results showed that
time spent on taskq_thread() was reduced by about 11%.
Apart from getting performance benefits, the number of threads in OSv was
nicely reduced (from ~344 threads to ~224; so possibly saving a good amount
of memory footprint).
Also good for stepping towards our synchronicity with ZFS upstream.

Addressing the issue #247.

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
jvmballoon: fix _soft_max_balloons check

Number of to be released balloons is calculated as a difference between
current number of balloons and sof max. If they are equal no balloons
are released and the loop repeats.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
sched: fix waitqueue race causing failure to wake up

When waitqueue::wake_all() wakes up waiting threads, it calls
sched::thread::wake_lock() to enqueue those waiting threads on the mutex
protecting the waitqueue, thus avoiding needless contention on the mutex.
However, if a thread is already waking, we let it wake naturally and acquire
the mutex itself.

The problem is that the waitqueue code (wait_object<waitqueue>::poll())
examines the wait_record it sleeps on and see if it has woken, and if not,
goes back to sleep.  Since nothing in that thread-already-awake path clears
the wait_record, that is what happens, and the thread stalls, until a timeout
occurs.

Fix by clearing the wait record.  As it is protected by the mutex, no
extra synchronization is needed.

Observed with iperf -P 64 against the guest.  Likely triggered by net channels
waking up the thread, and then before it has a chance to wake up, a FIN
packet arrives that is processed in the driver thread; so when the packets
are consumed the thread is in the waking state.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
mgmt: update to latest

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
vfs: fix partial non-blocking write

Our read() and write(), and their variants (pread, pwrite, readv, writev,
preadv, pwritev) all shared the same bug when it comes to a partial read or
write: they returned EWOULDBLOCK (EAGAIN) instead of returning successfully
with the number of bytes actually written or read, as they should have.

In the internals of the BSD read and write operations (e.g., sosend_generic)
each operation returns *both* an error number and a number of bytes left.
But at the end, the system call is expected to return just one of them -
either an error *or* a number of bytes. The existing read()/write() code,
when it saw the internals returning an error code, always returned it and
ignored the number of bytes. This was wrong: When the error is EWOULDBLOCK
and the number of bytes is non-zero, we should return this number of bytes
(i.e., a successful partial write), *not* the EWOULDBLOCK error.

This bug went unnoticed almost since the dawn of OSv, because partial reads
and writes are not common. For example, a write() to a blocking socket will
always return after the entire write is successful, and will not partially
succeed. Only when we write to an O_NONBLOCK socket, will it be possible to
see a partial write - But even then, we would need a pretty large write()
to see it only partially succeeding.

But this bug is very noticable when running the Jetty Web server (see issue
At some point it's like the response was restarted (complete with a second
copy of the headers). In Jetty's demo this was seen as half-shown images,
as well as corrupt output when fetching large text files like /test/da.txt.

Turns out that Jetty sends static responses in a surprisingly efficient
(for Java code...) way, using a single system call for the entire response:
It mmap()s the file it wishes to send, and then uses one writev() call to
send two arrays: The HTTP headers (built in malloc()ed memory), and the
file itself (from mmapped memory). So Jetty tries to write even a 1MB file
in one huge writev() call. But there's an added twist: It does so with the
socket configured to O_NONBLOCK. So for large writes, the write will only
partially succeed (empirically, only about 50KB will succeed), and Jetty
will notice the partial write and continue writing the rest - until the
whole file is sent. With the bug we had, part of the request will have been
written, but Jetty still thought the write didn't write anything so it would
start writing again from the beginning - causing the weird sort of response
corruption we've been seeing.

This patch also includes a test case which confirms this bug, and its fix.
In this test (tst-tcp-nbwrite), two threads communicate over a TCP socket
(on the loopback interface), one thread write()s a very large buffer and
the other receives what it can. We try this two times - once on a blocking
socket and once on a non-blocking socket. In each case we expect the number
of bytes written by one thread (return from write()) and the number read
by the second thread (return from read()) to be the same. With the bug we
had, in the non-blocking case we saw write() returning -1 (with
errno=EWOULDBLOCK) but read returned over 50,000 bytes, causing the test
to fail.

Fixes #257.

Signed-off-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
release-ec2: trivial: help screen enhancements

Help screen enhanced to make parameters syntax clear

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
release-ec2: Introduce AWS regions list parameter

Parameter --override-regions added

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
release-ec2: Base released images on Linux AMI

Linux AMI suits OSv better because of default settings
instances inherit on creation - mainly preconfigured SSH
access and open SSH ports.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
upload-ec2: Introduce script for images upload

Introduce script upload-ec2.sh that gets prebuilt
OSv images from S3 and converts them to EC2 AMIs.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Merge branch 'ec2-release-scripts'
Bump up apps submodule HEAD

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
run.py: allow multiple arguments to --trace

Allow to pass --trace multiple times to run.py so that the command
line interface is the same as for OSv's loader, eg:

  scripts/run.py --trace sched_wait* --trace memory_*

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
libc: Add program_invocation_name variables
 
Redis relies on the variables being present and correctly set.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
pthread_at_fork: bogus implementation

Because we do not support fork, there is no need to do anything upon fork (and if
it changes in the future, we should revisit this)

So we can get away with this simple implementation that just returns 0.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
pthread: stub setcanceltype

The implementation won't do anything anyway until we have setcancelstate().

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
mmu: correctly calculate size for unmap operations

Right now, we graciously accept - as does Linux - a mmap call for a size
smaller than a page: we align it up, and serve it. But the same alignment is
missing from unmap: so if the user rightfully tries to unmap using the same
size, it will fail.

The following test program succeeds on Linux but fails on OSv:

    int main () {
        void *ret = mmap(NULL, 64, PROT_READ, MAP_ANON | MAP_PRIVATE, -1, 0);
        if (ret == NULL) {
            return 1;
        }
        return munmap(ret, 64);
    }

After this patch, it works for us as well.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
tests: enhance mmap test

We have recently discovered a bug through which we fail to unmap a valid region.
This is fixed now, and this patch adds the failing condition to the test suite.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
pthread: implement mutexattr functions

redis (jemalloc to be precise) actually expects that functions to succeed.
Returning -1 here means it will not initialize correctly its memory allocator.

This simple implementation just stores the desired value, but does not change
anything in the underlying mutex.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>