migration: fix rate limiting

buffered_rate_limit is called to prevent the RAM migration callback
from putting too much data in the buffer.  So it has to check against
the amount of data currently in the buffer, not against the amount
of data that has been transferred so far.

s->bytes_xfer is used to communicate between successive calls of
buffered_put_buffer, with buffered_rate_tick resetting it every
now and then to prevent moving too much buffered data to the socket
at once.  But it does not matter for the producer of the data.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
exec: change ramlist from MRU order to a 1-item cache

Most of the time, only 2 items will be active (from/to for a string operation,
or code/data).  But TCG guests likely won't have gigabytes of memory, so
this actually goes down to 1 item.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
exec: change RAM list to a TAILQ

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
exec: sort the memory from biggest to smallest

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
exec: add a version number to ram_list

This will be used to detect if last_block might have become invalid
across different calls to ram_save_live.

Signed-off-by: Umesh Deshpande <udeshpan@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
protect the ramlist with a separate mutex

Add the new mutex that protects shared state between ram_save_live
and the iothread.  Writes need to take both locks (with the iothread
lock outside).  Reads can take either lock.

Signed-off-by: Umesh Deshpande <udeshpan@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
buffered_file: Move from using a timer to use a thread

We still protect everything except the wait with the iothread lock.
But we moved from a timer to a thread.  Steps one by one.

We also need to detect when we have finished with a variable "complete".

Signed-off-by: Juan Quintela <quintela@redhat.com>
migration: make qemu_fopen_ops_buffered() return void

We want the file assignment to happen before the thread is created to
avoid locking, so we just do it before creating the thread.

Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Orit Wasserman <owasserm@redhat.com>
migration: stop all cpus correctly

FIXME: wrong commit message

You can only stop all cpus from the iothread or an vcpu.  As we want
to do it from the migration_thread, we need to do this dance with the
botton handlers.

This patch is a request for ideas.  I can move this function to cpus.c, but
wondered if there is an easy way of doing this?

Signed-off-by: Juan Quintela <quintela@redhat.com>
migration: make writes blocking

Move all the writes to the migration_thread, and make writings
blocking.  Notice that are still using the iothread for everything
that we do.

Signed-off-by: Juan Quintela <quintela@redhat.com>
migration: remove unfreeze logic

Now that we have a thread, and blocking writes, we don't need it.

Signed-off-by: Juan Quintela <quintela@redhat.com>
migration: take finer locking

Instead of locking the whole migration_thread inside loop, just lock
migration_fd_put_notify, that is what interacts with the rest of the
world.

Signed-off-by: Juan Quintela <quintela@redhat.com>
buffered_file: Unfold the trick to restart generating migration data

This was needed before due to the way that the callbacks worked.

Signed-off-by: Juan Quintela <quintela@redhat.com>
buffered_file: don't flush on put buffer

We call buffered_put_buffer with iothread held, and buffered_flush() does
synchronous writes.  We only want to do the synchronous writes outside.

Signed-off-by: Juan Quintela <quintela@redhat.com>
buffered_file: unfold buffered_append in buffered_put_buffer

It was the only user, and now buffered_put_buffer just do the append

Signed-off-by: Juan Quintela <quintela@redhat.com>
savevm: New save live migration method: pending

Code just now does (simplified for clarity)

    if (qemu_savevm_state_iterate(s->file) == 1) {
       vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
       qemu_savevm_state_complete(s->file);
    }

Problem here is that qemu_savevm_state_iterate() returns 1 when it
knows that remaining memory to sent takes less than max downtime.

But this means that we could end spending 2x max_downtime, one
downtime in qemu_savevm_iterate, and the other in
qemu_savevm_state_complete.

Changed code to:

    pending_size = qemu_savevm_state_pending(s->file, max_size);
    DPRINTF("pending size %lu max %lu\n", pending_size, max_size);
    if (pending_size >= max_size) {
        ret = qemu_savevm_state_iterate(s->file);
     } else {
        vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
        qemu_savevm_state_complete(s->file);
     }

So what we do is: at current network speed, we calculate the maximum
number of bytes we can sent: max_size.

Then we ask every save_live section how much they have pending.  If
they are less than max_size, we move to complete phase, otherwise we
do an iterate one.

This makes things much simpler, because now individual sections don't
have to caluclate the bandwidth (it was implossible to do right from
there).

Signed-off-by: Juan Quintela <quintela@redhat.com>
migration: include qemu-file.h

They don't use/know anything about buffered-file.

Signed-off-by: Juan Quintela <quintela@redhat.com>
migration-fd: remove duplicate include

Signed-off-by: Juan Quintela <quintela@redhat.com>
memory: introduce memory_region_test_and_clear_dirty

This function avoids having to do two calls, one to test the dirty bit, and
other to reset it.

Signed-off-by: Juan Quintela <quintela@redhat.com>
ram: Use memory_region_test_and_clear_dirty

This avoids having to do two walks over the dirty bitmap, one reading
the dirty bits, and another cleaning them.

Signed-off-by: Juan Quintela <quintela@redhat.com>