mmu: fix how mappings into ARC buffer are tracked

Currently all mapping are keyed on ARC buffer start when mapping is
added, but on remove pointer into ARC buffer is used, so remove may
leave no longer valid mappings in the database. This patch fixes it
by using a pointer into ARC as a key, the same pointer that is used
during removal.

Signed-off-by: Gleb Natapov <gleb@cloudius-systems.com>
Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
zfsbuffers: Do not truncate files

There is a problem with the way ZFS currently handles its buffers, which is
actually a limitation of our allocator: buffers smaller than a page won't be
page aligned even if we ask for it. Therefore, if the buffer we are mapping
falls into this category, we will map the wrong location.

The way I solved this problem was so stupid, that in retrospect I can't even
believe I did it: when the file would run out of size, we would truncate the
file. This is obviously wrong because reading a file is not expected to change
its size in any circumstance, and if anybody relied in the actual size, we will
be crashing something. This is the bug that plagued Cassandra.

Not truncating, however, brings back the original problem. One solution I have
considered is to always allocate at least a page for data allocations (leaving
metadata alone), but that would deviate from ZFS and harm many-small-files
workloads.

However, During testing, I have noticed though that ZFS will allocate small
buffers only when the file itself is small. This means that we can just avoid
using the special shared mapping for small files - which makes sense anyway.

For instance, if we have a file that is 128k + 1byte (remember 128k is ZFS's
maximum buffer size), both buffers will be large enough to be aligned. And if I
that ever fails to hold, we will now see an assertion hit instead of a random
bug. In time, we should fix our allocator to provide alignment guarantees.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
tests: improve zfs shared buffers tests

We have seen bugs with mmap shared/file handling for small files. This patch tests
some of the corner scenarios to find those problems.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
zfsbuffers: kill bogus variable

Spotted by code review. Gleg had spotted one improper use of "i", but
there was another. In this case we iterate over nothing, and i is always 0.
It is uninitialized to begin with, and the code works just because it is
being set to 0 by luck.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
zfsbuffers: reference count the arc buffer

Gleb has noticed that the ARC buffers can go unshared too early. This will
happen because we call the UNMAP operation on every put(). That is certainly
not what we want, since the buffer only has to be unshared when the last
reference is gone.

Design decisions:
1) We obviously can't use the arc natural reference count for that, since
bumping it would make the buffer unevictable.
2) We could modify the arc_buf structure itself to add another refcnt (minimum
4 bytes).  However, I am trying to keep core-ZFS modifications to a minimum,
and only to places where it is totally unavoidable.

Therefore, the solution is to add another hash, which will hash the whole
buffer instead of the physaddr like the one we have currently. In terms of
memory usage, it will add only 8 bytes per buffer (+/- 128k each buffer), which
makes for a memory usage of 64k per mapped Gb compared to the arc refcount
solution. This is a good trade off.

I am also avoiding adding a new vop_map/unmap style operation just to query the
buffer address from its file attributes (needed for the put side). Instead, I
am conventioning that an empty iovec means query, and a filled iov means
unshare.

Signed-off-by: Glauber Costa <glommer@cloudius-systems.com>
Merge branch 'glommer/zfsbuffers' of github.com:glommer/osv
scripts/test.py: Blacklist tst-dns-resolver.so

The tst-dns-resolver.so fails spuriously. Blacklist it until the problem
is fixed to keep Jenkin builds running.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
manifest: Add /etc/mnttab from the upload manifest process

/etc/mnttab is required by libzfs to get running properly, so let's
create it as an empty file.

ryao from zfsonlinux and openzfs told me that an empty /etc/mnttab is used
on Linux. Also reading the libzfs code shows that /etc/mnttab mostly used for
management of the file itself, nothing that would prevent some libzfs
functionality from working.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
drivers: Add zfs device to allow use of zfs commands

Previously, zfs device was being only provided to allow the use of
commands needed to create the zpool, and so the file system.
At that time, doing so was quite enough, however, making zfs
device, i.e. /dev/zfs part of every OSv instance would allow us
to use commands that will help analysing, debugging, tuning
the zpool and file systems there contained.

The basic explanation is that those commands use libzfs which in
turn relies on /dev/zfs to communicate with the zfs code.

Commands example:
zpool, zfs, zdb. The latter one not being ported to OSv yet.
This patch will also be helpful for the ongoing ztest porting.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
mkfs: Switch to the newly created zfs device driver

Remove the embedded driver, and start using the one from drivers
sub-system. Still relies on the manual creation of /etc/mnttab as
the upload manifest wasn't yet processed.

Reviewed-by: Nadav Har'El <nyh@cloudius-systems.com>
Signed-off-by: Raphael S. Carvalho <raphaelsc@cloudius-systems.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
scripts: Python3 support for scripts/silentant.py

Add explicit conversion form bytes array to string which returned by
Popen.communicate().

Use Python3 style print function.

Signed-off-by: Zifei Tong <zifeitong@gmail.com>
Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
tests: test various buffer sizes in misc-bdev-write test

Xen block driver now supports new featute called indirect descriptors.
This feature allows to put more data into each ring cell but it activates
for "long" reads and writes only - longer than 11 pages.

With this patch test by default runs 2 scenarious:
  * 1 page buffers
  * 32 pages buffers

Also introduced command line parameter to specify
size of buffers explicitly.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
tests: introduce write/read/verify test

Test misc-bdev-rw introduced.

The test writes buffers of various lengths to block device,
reads data back and verifies data read is the same as data written.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
blkfront: introduce functions for device features detection

This is a refactoring commit that isolated some xenstore access
logic to make it reusable

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
blkfront: isolate command data grant references aquisition

This is a refactoring commit to simplify future indirect descriptors code.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
blkfront: isolate commands allocation

This is a refactoring commit to simplify future indirect descriptors code.

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
blkfront: introduce indirect descriptors support

Indirect descriptors for Xen PV disks is a newly
introduced protocol feature that increases
block devices performance in case of long transfers.

Detailed description:
http://blog.xen.org/index.php/2013/08/07/indirect-descriptors-for-xen-pv-disks/

Measurement results (RAM-backed storage):

Xen w/o indirect descriptors:

1046.251 Mb/s
1097.931 Mb/s
1149.672 Mb/s
1137.696 Mb/s
1135.481 Mb/s
1249.163 Mb/s
1115.417 Mb/s
1118.063 Mb/s
Wrote 11344.750 MB in 10.00 s = 1134.165 Mb/s

Xen w/ indirect descriptors:

bdev-write test offset limit: 250000000 byte(s)
1234.715 Mb/s
1360.013 Mb/s
1323.663 Mb/s
1336.916 Mb/s
1342.617 Mb/s
1332.882 Mb/s
1302.094 Mb/s
Wrote 13233.250 MB in 10.04 s = 1318.639 Mb/s

KVM:

bdev-write test offset limit: 250000000 byte(s)
674.729 Mb/s
698.736 Mb/s
627.677 Mb/s
630.209 Mb/s
742.977 Mb/s
759.205 Mb/s
681.476 Mb/s
655.717 Mb/s
713.231 Mb/s
688.478 Mb/s
Wrote 6872.750 MB in 10.00 s = 687.124 Mb/s

Signed-off-by: Dmitry Fleytman <dmitry@daynix.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
support vga cursor

Add cursor update callback to libtsm, update cursor position register on VGA device using the callback.
Fixes #220.

Signed-off-by: Takuya ASADA <syuu@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
scripts: Add script for building a Capstan image

This adds a script for creating Capstan compatible images that can be
uploaded to a S3 bucket for distribution.

To use it to build a base image, for example, type:

  $ ./scripts/build-capstan-img cloudius/osv-base empty "OSv base image"

which builds a Capstan image named "cloudius/osv-base" with "make
image=empty" and places QEMU and VirtualBox images under
"build/capstan/cloudius/osv-base" together with an index file.

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>
scripts: Add script for building OSv release

This adds a script for building a Capstan repository for an OSv release
with the following images:

  - OSv core (no API, no CLI)
  - Default (with API, CLI)
  - Cassandra virtual appliance (with API)
  - Tomcat virtual appliance (with API)
  - Iperf virtual appliance (with API)

Signed-off-by: Pekka Enberg <penberg@cloudius-systems.com>
Signed-off-by: Avi Kivity <avi@cloudius-systems.com>