net: don't close the tcp connection within tcp_do_segment()

tcp_do_segment() has a nasty habit of sometimes closing the tcp connection.
This plays havoc with the locking rules (as updated for net channels), since
tcp_do_segment() is supposed to return with the lock held, yet tcp_close()
will unlock if the connection is disassociated from a socket.

Fix by having tcp_do_segment() not close the connection, and instead indicate
its desire to do so to its caller.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
virtio: switch to page-sized external buffers

Curently virtio uses MCLBYTES (2KB) buffers for virtio.  This hits a weakness
in our UMA and memory allocation implementations:

- UMA appends a 4-byte reference count to the allocation, resulting in an
  2052 byte allocation request
- Our allocator only services requests up to 2048 bytes from the mempool
  allocator; larger requests are serviced by the large allocator, which is
  significately slower, and is also serialized by a mutex.

Switch to an EXT_EXTREF mbuf with the buffer and refcount allocated
separately.  This also allows us to switch to 4K allocations serviced by
the page allocator (memory::alloc_page()), which is our fastest allocator,
and halves the lengths of the mbuf chains on streaming workloads.

Increases iperf stream (host->guest) results from ~12Gbps to ~34Gbps on my
desktop.

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Move addr_range class to a separate file

It will be used outsize of mmu code.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
mmu: delegate file_vma creation to file class

Some kind of files may want to handle mmap differently. By delegating
file_vma creation to the file class it is possible for each file to
create customized vma.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
mmu: rename map_page_ops related structures

map_page_ops is used by page mapper to managed individual page
allocations and freeing, but the name is not reflects that.
page_allocator describes better the purpose of the structure.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
mmu: provide page_allocator specialization for files that support mmap semantic

Implement page_allocator specialization that delegates page
allocation/deallocation to mmaped file. It will be used to mmap files
that support get_page/put_page calls. Others fall back to old way of
reading file.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
mmu: implement shared memory file

shmfd is a pseudo file that allows sharing of anonymous memory. Each
mapping for this file points to the same memory. Only huge pages are
supported for now.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
mmu: mark address parameter as const

shmdt() is defined to get const pointer. Next patch implements shmdt
using unmap(), so const qualifier either need to be casted off or
unmap() and friends need to get const pointer as a parameter too. This
patch implement the later.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
libc: implement System V shared memory interface

Implement shmget/shmat/shmdt/shmctl function required by System V shared
memory interface.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
tests: add System V shared memory test

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Merge branch 'shm' of https://github.com/gleb-cloudius/osv

"
System V shared memory implementation

It looks like fio needs it although with single address space any memory
is shared anyway.
"

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
gdb: fix conversion of gdb.Value to int

Python3 no longer has "long" type, and what was "long" on python2 is
now "int". To make the code portable commit
eb5eb91041bb6316b29c38679a4d575d40e860ed switched long() conversions
to int() conversion, which should be equivalent. It turns out that
GDB's implementation of gdb.Value, when compiled against python 2, has
a slight difference between int() and long() conversions. The long
conversion can be used on pointers whereas int conversion cannot. I
don't about any reason it should be like this, maybe it is a GDB bug:

This is int() handler:

      if (!is_intlike (type, 0))
	error (_("Cannot convert value to int."));
      l = value_as_long (value);

This is long() handler:

      if (!is_intlike (type, 1))
	error (_("Cannot convert value to long."));
      l = value_as_long (value);

Note the second argument to is_intlike() differs between variants, "1"
means "pointers allowed". Since on python3 int() conversion calls the
long handler, it works as expected. As a workaround for this I defined
to_int() function which performs the conversion in a portable way. This
needs to be done only when the value may be a pointer.

Places which check:

 if int(x) != 0:

can be simplified to:

 if x:

which works with both pointers and numbers in both versions.

Reported-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Merge branch 'net_channel'

tcp_close() related bug fix
switch virtio to page buffers

Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
mkzfs.py: Use less memory during image build

Fixes #198.

Signed-off-by: Sanidhya Kashyap <sanidhya.k@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
upload_manifest.py: Use less memory during image build

Fixes #198.

Signed-off-by: Sanidhya Kashyap <sanidhya.k@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
route: Add host route support

Add similar function of the following route command
in Freebsd to OSv.

   $ route add -host $host_ip -interface $interface_name
   e.g.
   $ route add -host 10.240.0.1 -interface eth0

This is useful when the gateway is not reachable within the network
specified by the interface. For instance, GCE's DHCP server provides:

   ip=10.240.10.15, netmask=255.255.255.255, gw=10.240.10.1

A host route entry is needed before we can add the default route entry.

With this patch, the route table looks like:

   Destination         Gateway             Flags  Netif
   default             10.240.0.1          UGS    eth0
   10.240.0.1          42:01:0a:f0:e0:c5   UHS    eth0
   10.240.10.15        link#2              U      eth0
   10.240.10.15        link#2              UHS    lo0
   127.0.0.1           link#1              UH     lo0

Signed-off-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
build.mk: unify boost-tests list construction method

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Tomasz Grabiec <tgrabiec@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
run.py: Fix API user networking port forwarding

The API is listening to port 8000, not 8080.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
virtio-net: fix handling of bad buffers

When we discard a bad buffer, we don't advance to the next one, so we'll free
it again on the next pass, resulting in a double-free.

Tested-by: Asias He <asias@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Merge branch 'master' into aarch64-next

Conflicts:
	scripts/mkzfs.py
	scripts/upload_manifest.py