scripts: add EC2 metadata service simulator

Run it like this:

  $ sudo scripts/ec2-simulator.py

The tool starts HTTP server which can be reached via standard cloud
meta-data service IP (169.254.169.254).

It needs to modify NAT rules and insert an IP address so it needs to be
run with "sudo".

You can set user data via command line arguments, eg:

  $ sudo scripts/ec2-simulator.py -f user-data.yaml

Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
virtio: fix missed notification

In virtio::vring::kick(), we read avail_event (which is written by the host)
in order to see if we need to notify the host about our last write.  However,
the processor may speculate this read, moving it before the avail_idx write
(in virtio::vring::add_buf()), in which case we will see a stale value.

Insert a memory fence in order to ensure we read a fresh value.

The fixes throughput issues in request/repsonse benchmarks, esp. netperf
TCP_RR.

Fixed #383.

Reviewed-by: Dor Laor <dor@cloudius-systems.com>
Reviewed-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
build: teach osv-version.sh not to pick up random tags

Otherwise the build blows up when some regex doesn't match.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
tests: fix tst-dlfcn

info.dli_sname can be null; account for it.

Suggested by Nadav.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
tests: reduce rcu hashtable test running time

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
EC2 simulator: Add metadata API for Cassandra autoconfig

This adds more metadata APIs that are needed by Cassandra's EC2 cluster
auto-configuration service called Datastax Reflector.

Reviewed-by: Tomasz Grabiec <tgrabiec@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
cli: Refactoring command structure

A small needed refactoring, hopefully before CLI gains attention upstream.

* Refactor the command structure to match the Lua suggested module structure.
* Change "run" method name to "main".

Signed-off-by: Or Cohen <orc@fewbytes.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
cli: Add "hello" boilerplate command example

Signed-off-by: Or Cohen <orc@fewbytes.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
vfs: implement sendfile()

An implementation for sendfile() using mmap() and write().
It copies data from in_fd to out_fd without any intermediates.
It assumes that if mmap() succeeds, then munmap() will most
likely succeed. If not, it ignores return value of munmap().

Fixes #363
Fixes #273

Reviewed-by: Glauber Costa <glommer@cloudius-systems.com>
Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Jaspal Singh Dhillon <jaspal.iiith@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
tests: add tests for sendfile()

A battery of tests for sendfile().
The tests belong to 2 groups:
1. copy data from file to file
2. copy data from file to a socket
In both cases, two sub-cases arise: the offset is/is not NULL.

Signed-off-by: Jaspal Singh Dhillon <jaspal.iiith@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
ide: fix drive availability check

Nadav explains:

  After commit 2a56d98d7de1db014d8ac03cca12ecfb524dd83c fixed the PCI
  offset constants, OSv suddenly became able to mount IDE devices.
  Unfortunately, on VMware, it wrongly recognized both an IDE and a pscsi
  device, resulting in two drivers trying to take the "vblk0" name and
  causing a sys_panic() and a crashed boot (see full stack trace in issue
  #369).

  It turns out that our IDE device recognition code wasn't diligent
  enough in trying to recognize devices which appear to be there, but
  actually aren't, as happened on VMware. Remember that a physical
  computer can have an IDE controller, but no disk actually plugged in,
  and so can the virtual counterpart. This patch adds better tests for the
  existence of the IDE disk, and now on VMware we no longer wrongly-see an
  IDE disk, and VMware boot is back to normal.

Fixes #369

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
tests: run tests with --unsafe-cache

Much faster on spinning disks.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Test tst-poll.cc can not be runned twice

Test tst-poll throws an exception on second execution:
../tst-poll.cc(78): fatal error in "test_polling_on_one_socket":
critical check bind(s, (struct sockaddr *) &laddr, sizeof(laddr)) == 0
failed. Test test_polling_on_one_socket left the binded socket open -
cuasing the second execution to fail on bind with EADDRINUSE (98)
/* Address already in use */

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Shlomi Livne <shlomi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
xen blkfront: Don't for quiesce on absent flush

BSD seems to do that in an upper layer that we lack, and Linux also just reports
flush as not supported. Barriers are a different story, and will be handled in a
different patch.

Tested-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
xen blkfront: software barrier support

This code comes largely from BSD's review r252260, with modifications
from our infrastructure.

After this patch, filesystem requests that ask for serialization will
freeze the request queue until that command is processed, unless we have
Hypervisor support for serialization. That should be present in any
modern Xen hypervisor, but it is not present on EC2.

Tested-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
percpu_xmit: Use a proper memory order in clear_pending()

std::memory_order_release -> std::memory_order_seq_cst
We need to ensure that clear_pending() is not reordered with _mg.pop()
in poll_until().

We use a "relaxed" ordering in the clear_pending() itself (renamed to
clear_pending_weak()) and if _mg.pop() returns FALSE we put an
std::atomic_thread_fence(std::memory_order_seq_cst) and call for
_mg.pop() again to ensure the proper ordering.

Signed-off-by: Vlad Zolotarov <vladz@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
vfs: fix errno in sendfile()

The man-page of sendfile() says to return the following on errors:

EBADF  The input file was not opened for reading or the output file was
       not opened for writing.

EINVAL Descriptor is not valid or locked, or an mmap(2)-like operation
       is not available for in_fd.

But sendfile() on Linux returns EBADF when the descriptor is not valid.
It return EINVAL when writing to the file has failed.  Change sendfile()
to behave the same as Linux.

Also, replace C style code with C++ such as using mmu::page_size instead
of PAGESIZE.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Jaspal Singh Dhillon <jaspal.iiith@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
tests: fix tst-sendfile.cc

Fix test to expect EBADF on a bad fd.
Use std::thread to allow the test to be compiled and run on linux.
Move the setup part (socket(),bind(),listen()) of server outside the thread.

Signed-off-by: Jaspal Singh Dhillon <jaspal.iiith@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
vfs: increase vnode refcount under vnode_lock lock in dentry_alloc()

v_refcnt is protected by vnode_lock and should not be accessed without
holding it.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
pthread: implement pthread yield

trivial wrapper around sched::yield

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>