migration: add an indicator for bulk state of ram migration

the first round of ram transfer is special since all pages
are dirty and thus all memory pages are transferred to
the target. this patch adds a boolean variable to track
this stage.

Signed-off-by: Peter Lieven <pl@kamp.de>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
migration: do not sent zero pages in bulk stage

during bulk stage of ram migration if a page is a
zero page do not send it at all.
the memory at the destination reads as zero anyway.

even if there is an madvise with QEMU_MADV_DONTNEED
at the target upon receipt of a zero page I have observed
that the target starts swapping if the memory is overcommitted.
it seems that the pages are dropped asynchronously.

this patch also updates QMP to return the number of
skipped pages in MigrationStats.

Signed-off-by: Peter Lieven <pl@kamp.de>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
migration: do not search dirty pages in bulk stage

avoid searching for dirty pages just increment the
page offset. all pages are dirty anyway.

Signed-off-by: Peter Lieven <pl@kamp.de>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
migration: use XBZRLE only after bulk stage

at the beginning of migration all pages are marked dirty and
in the first round a bulk migration of all pages is performed.

currently all these pages are copied to the page cache regardless
of whether they are frequently updated or not. this doesn't make sense
since most of these pages are never transferred again.

this patch changes the XBZRLE transfer to only be used after
the bulk stage has been completed. that means a page is added
to the page cache the second time it is transferred and XBZRLE
can benefit from the third time of transfer.

since the page cache is likely smaller than the number of pages
it's also likely that in the second round the page is missing in the
cache due to collisions in the bulk phase.

on the other hand a lot of unnecessary mallocs, memdups and frees
are saved.

the following results have been taken earlier while executing
the test program from docs/xbzrle.txt. (+) with the patch and (-)
without. (thanks to Eric Blake for reformatting and comments)

+ total time: 22185 milliseconds
- total time: 22410 milliseconds

Shaved 0.3 seconds, better than 1%!

+ downtime: 29 milliseconds
- downtime: 21 milliseconds

Not sure why downtime seemed worse, but probably not the end of the world.

+ transferred ram: 706034 kbytes
- transferred ram: 721318 kbytes

Fewer bytes sent - good.

+ remaining ram: 0 kbytes
- remaining ram: 0 kbytes
+ total ram: 1057216 kbytes
- total ram: 1057216 kbytes
+ duplicate: 108556 pages
- duplicate: 105553 pages
+ normal: 175146 pages
- normal: 179589 pages
+ normal bytes: 700584 kbytes
- normal bytes: 718356 kbytes

Fewer normal bytes...

+ cache size: 67108864 bytes
- cache size: 67108864 bytes
+ xbzrle transferred: 3127 kbytes
- xbzrle transferred: 630 kbytes

...and more compressed pages sent - good.

+ xbzrle pages: 117811 pages
- xbzrle pages: 21527 pages
+ xbzrle cache miss: 18750
- xbzrle cache miss: 179589

And very good improvement on the cache miss rate.

+ xbzrle overflow : 0
- xbzrle overflow : 0

Signed-off-by: Peter Lieven <pl@kamp.de>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Add QemuFileWritevBuffer QemuFileOps

This will allow us to write an iovec

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Add socket_writev_buffer function

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Update bytes_xfer in qemu_put_byte

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Store the data to send also in iovec

All data is still copied into the static buffer.
Adjacent iovecs are coalesced so we send one big buffer
instead of many small buffers.

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Use writev ops if available

Update qemu_fflush and stdio_close to use writev ops if they are available
Use the buffers stored in the iovec.

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Add qemu_put_buffer_async

This allows us to add a buffer to the iovec to send without copying it
into the static buffer, the buffer will be sent later when qemu_fflush is called.

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Use qemu_put_buffer_async for guest memory pages

This will remove an unneeded copy of guest memory pages.
For the page header and device state we still copy the data to the
static buffer the other option is to allocate the memory on demand
which is more expensive.

Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Merge remote-tracking branch 'luiz/queue/qmp' into staging

# By Corey Bryant (2) and others
# Via Luiz Capitulino
* luiz/queue/qmp:
  New QMP command query-cpu-max and HMP command cpu_max
  qmp: fix handling of boolean values in qmp-shell
  QMP: TPM QMP and man page documentation updates
  QMP: Remove duplicate TPM type from query-tpm
virtio-scsi: don't use pointer for configuration.

The configuration field must not be a pointer as it will be used for virtio-scsi
properties. So *conf is replaced by conf.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Tested-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Message-id: 1363875320-7985-2-git-send-email-fred.konrad@greensocs.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
virtio-scsi: allocate cmd_vqs array separately.

Allocate/Free the cmd_vqs array separately to have a fixed size device.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Tested-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Message-id: 1363875320-7985-3-git-send-email-fred.konrad@greensocs.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
virtio-scsi: moving host_features from properties to transport properties.

host_features field is part of the transport device. So move all the
host_features related properties into transport device.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Tested-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Message-id: 1363875320-7985-4-git-send-email-fred.konrad@greensocs.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
virtio-scsi: add the virtio-scsi device.

Create virtio-scsi which extends virtio-device, so it can be connected on
virtio-bus.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Tested-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Message-id: 1363875320-7985-5-git-send-email-fred.konrad@greensocs.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
virtio-scsi-pci: switch to new API.

Here the virtio-scsi-pci is modified for the new API. The device virtio-scsi-pci
extends virtio-pci. It creates and connects a virtio-scsi during the init.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Tested-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Message-id: 1363875320-7985-6-git-send-email-fred.konrad@greensocs.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
virtio-scsi-s390: switch to the new API.

Here the virtio-scsi-s390 is modified for the new API. The device
virtio-scsi-s390 extends virtio-s390-device as before. It creates and
connects a virtio-scsi during the init. The properties are not modified.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Tested-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Message-id: 1363875320-7985-7-git-send-email-fred.konrad@greensocs.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
virtio-scsi-ccw: switch to new API

Here the virtio-scsi-ccw is modified for the new API. The device
virtio-scsi-ccw extends virtio-ccw-device as before. It creates and
connects a virtio-scsi during the init. The properties are not modified.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Tested-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Message-id: 1363875320-7985-8-git-send-email-fred.konrad@greensocs.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
virtio-scsi: cleanup: use QOM casts.

As the virtio-scsi-pci and virtio-scsi-s390 are switched to the new API,
we can use QOM casts.

Signed-off-by: KONRAD Frederic <fred.konrad@greensocs.com>
Reviewed-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Tested-by: Cornelia Huck <cornelia.huck@de.ibm.com>
Message-id: 1363875320-7985-9-git-send-email-fred.konrad@greensocs.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>