virt: Adds remote commander package to virttest libs

The remote commander consists of 3 parts messenger,
remote_master and remote_runner (python shell).

The messenger is base class for remote_master and remote_runner
which is used for sending picked data over IO. IOs are implemented
like wrappers to special type of IO. For example python streams,
file descriptors, paramiko class. ...

data => cPickle.dump(data) => IO_wrapper_OUT.write()
IO_wrapper_IN.read(len, timeout) => cPickle.load(data) => data

messanger could be used for general communication over pipe
or similar IO interface.

The remote_master is responsible for passing commands to
remote_runner and receiving results of commands.

example:
   remote_master.CommanderMaster(inw, outw, False)

where:
  inw is wrapper to input stream from remote_runner -> remote_master
  outw is wrapper to output stream from remote_master -> remote_runner
  False disables debug mode during which debug log are printed.

The remote_runner is responsible for starting commands
in system where the remote_runner running.

Command can be started in 4 modes:
1) blocking:
  Remote master wait until command on remote_runner is finished and
  after that it returns results.
   cmd = remote_master.CommanderMaster(inw, outw, False)
   a = cmd.shell("ip a")     # returns cmd
                shell("ip a") -> remote_runner
            remote_master     <- basecmd object
              wait            -> fork new process
              wait            -> wait till end of process
   a  <-    remote_master     <- results
   a.stdout   # stdout of command
   a.stderr   # stderr of command
   a.results  # what ever what command returns. (needs to be picleable).

2) async:
  It is possible start lots of async command on remote_runner
simultaneous.
   cmd = remote_master.CommanderMaster(inw, outw, False)
   a = cmd.async.shell("ip a")     # returns cmd
         async.shell("ip a") -> remote_runner
   a  <-    remote_master     <- basecmd object
             do whatever      -> fork new process
             do whatever      -> wait till end of process
   a.wait_response()   # wait for any response from remote command
                       # stdout, stderr, results
   a.wait()    # wait until command return result.
   a  <-    remote_master     <- results
   a.stdout   # stdout of command
   a.stderr   # stderr of command
   a.results  # what ever what command returns. (needs to be
pickleable).

3) nohup:
  It is possible start lots of nohup command on remote_runner
simultaneous.
  It is also possible loose connection with remote machine (kill
remote_runner)
  and after that reconnect to running nohup process.
  There could be problem with really long outputs from nohup process the
buffer
  in communication interface could overflow. Keep it in mind!!!
   cmd = remote_master.CommanderMaster(inw, outw, False)
   a = cmd.nohup.shell("ip a")     # returns cmd
          nohup.shell("ip a") -> remote_runner
   a  <-    remote_master     <- basecmd object
             do whatever      -> daemonize new process
                              -> create communication interface to nohup process.
             do whatever      -> wait till end of process

   stdin, stdout, stderr = ssh.exec_command("/usr/bin/env python /tmp/remote_runner.py agent", 0)

   #!!!!!!! reconnect to remote_runner interface.
   cmd = remote_master.CommanderMaster(ParamikoIOWrapperIn(stdout),
                                       ParamikoIOWrapperOut(stdin),
                                       False)
   #!!!!!!! set new commander for old commands.
   a.set_commander(cmd)       -> reconnect to nohup communication interface.
                              -> continue in communication.
   a.wait_response()   # wait for any response from remote command
                       # stdout, stderr, results
   a.wait()    # wait until command return result.
   a  <-    remote_master     <- results

   a.stdout   # stdout of command
   a.stderr   # stderr of command
   a.results  # what ever what command returns. (needs to be pickleable).

4) manage:
  It starts command in remote_runner process without forking.
  It could be dangerous but it is really useful for control of
remote_runner.
  For example it is possible start function cmd.manage.exit() for
exiting of remote_runner.
  Remote master wait until command on remote_runner is finished and
  after that it returns results.
   cmd = remote_master.CommanderMaster(inw, outw, False)
   a = cmd.manage.add_function  # return cmd
         manage.add_function(remote_master.get_source(func_name)) -> remote_runner
            remote_master     <- basecmd object
              wait            -> wait till end of process
   a  <-    remote_master     <- results
   b = cmd.async.func_name(args..)
   a.stdout   # stdout of command
   a.stderr   # stderr of command
   a.results  # what ever what command returns. (needs to be pickleable).

There are also prepared helper function in remote_runner.
   shell        # start command in shell (bash ... )
   interactive  # start interactive python shell
   send_msg     # sends msg to specific command
   add_function # for register class of function to global namespace of runner
                # if add_function is started in manage mode then all
                # commands
                # could use this new function and it is possible to call
                # the function
                # like normal command from remote_master.
   copy_file    # send file to remote system very naive implementation.
   import_src   # import python module to python. (copied file or python libs)
   exit         # kill the remote_runner.

Signed-off-by: Jiří Župka <jzupka@redhat.com>
virt: Creates control interface to aexpect process and add raw mode.

Adds new interface to aexpect process ctrlpipe.
Allows switch aexpect terminal to raw mode.
This changes allows use the terminal for binary communication.
Usable for program communication.

Signed-off-by: Jiří Župka <jzupka@redhat.com>
virt: Adds remote_commander method to virt_vm.

Allows start commands and collect results really easy way
in remote system.

import time
import logging
from autotest.client.shared import error

@error.context_aware
def run(test, params, env):
    """
    Qemu reboot test:
    1) Log into a guest
    3) Send a reboot command or a system_reset monitor command (optional)
    4) Wait until the guest is up again
    5) Log into the guest to verify it's up again

    :param test: QEMU test object
    :param params: Dictionary with the test parameters
    :param env: Dictionary with test environment.
    """

    timeout = float(params.get("login_timeout", 240))
    vm = env.get_all_vms()[0]
    vm.wait_for_login(timeout=timeout)

    cmd = vm.remote_commander()
    commands = []
    for _ in xrange(2):
        commands.append(cmd.async.shell("ip a"))

    for c in commands:
        logging.info(c.wait().stdout)

Signed-off-by: Jiří Župka <jzupka@redhat.com>
virttest.virt_vm: Remote runner copy speedup

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
virttest.env_process: Speed up remote login sessions cleanup code

Signed-off-by: Lucas Meneghel Rodrigues <lmr@redhat.com>
Merge pull request #1475 from autotest/gh-1449

Remote commander rebase