virt: Rewrite network/nic management

* Unattended installs default to lookup of netdst IP by name if
  url_auto_up param is not specified
+ Integrated new network code with kvm module
+ Removed net_type use and parameter b/c it was always synonomous with
  bridge networking.  Now, setting nettype=bridge should have the same
  behavior as before (bridge=virtnet0 net_type=tap).
+ libvirt_vm.start() setup of defined mac-addresses if not specified
+ libvirt_vm.start() issues warnings if requested mac differs from defined
+ Added many logging and debugging statements (some are temporary)
+ unit tests: New virt_utils_unittests tests
+ Updated shared ping test module
+ Updated kvm nic_bonding test module
+ Updated kvm nic_hotplug test module comment
+ virt_utils Env init now always warns when creating an empty env file
+ virt_utils Env init prints warning traceback when loading env fails
+ Pickling support for all new classes
+ Raise exception when adding nic with clashing name
+ New virt_utils_unittest deletes temporary testing database
+ IP/MAC cache is only updated with new entries
+ VM cleanup resets static addresses properly
+ Enhanced output of needs_restart() VM comparison function
+ Enhanced Env class to output more details on non-fatal load failures
+ Old virt_util functions all raise depreciation warnings
  to aide bug squashing
+ remove unnecessary nic_mode from subtests parameter, rely on default
    nettype=bridge instead
+ Implimented new nic/net management within libvirt_vm
+ Added stubbs for nic activation/deactivation in libvirt_vm
+ Removed extranious netdev_id/device_id/tap/ifname support from libvirt_vm
+ Enhanced logging for vm.needs_restart()
+ Changed libvirt vm to only free mac addresses for undefined VMs
+ Improved libvirt vm.is_persistent() to handle non-existing VMs
+ Defined mac address prefix convensions for libvirt (kvm/xen)
+ virt config: New universal virt. networking configuration
        nettype + netdst parameters determine networking style and
        destination for virtual NICs.  Separate NIC sets can be defined
        for each VM.  Documented examples of other per-nic parameters
        in config. comments.
+ virt_utils change etails:

New Class: PropCan
    Allow container-like access to fixed set of items (__slots__)
    This allows NIC/Net handling code to lock-down a fixed set of
    parameters relevant to a vm_type / driver_type combination.
    Data is accessible in a dictionary-like manner or as object
    properties, however empty or None value items will pretend to
    not exist.  The special empty/None handling optimises access
    and storage by only representing values containing meaningful
    data.

New Class: Base_VirtIface
    Networking information for single guest interface and host
    connection.  This is an abstract base-child-class of PropCan
    containing the minimal amount of networking keys needed, along
    with some basic name and mac address handling methods.  Some
    amount of effort is made to ensure stored mac address data
    is in an expected format - two ASCII character hex-byte
    representation, separated by :'s, six or less bytes in total.
    Complete mac addresses are generated by filling in missing
    trailing bytes from a sequential pool maintained as a class
    variable.  Numbering starts randomly from 0-255, then proceeds
    incrementally for each byte requested.  Sequential byte generation
    used to ease unit testing.

New Class: Libvirt_VirtIface & KVM_VirtIface
    Specializations of Base_VirtIface parent class (and PropCan)
    with networking keys specific to Libvirt or KVM respectively.
    If a new type or variation is needed, simply subclass Base_VirtIface
    and update the VMNet_Style.VMNet_Style_Map as needed.  The new or
    variant class will be used automatically based on resolving
    vm_type, and/or driver_type from the supplied Params dict (see
    other classes below)

New Class: VMNet
    Abstract base-class collection of networking information, stored
    within a particular container class (e.g. a subclass of
    Base_VirtIface). This class behaves like a list of PropCans.
    It may be initially empty or created from a list-like container
    of dictionary-like objects holding key/value pairs of networking
    data.  Items within a VMNet instance are indexed by number and
    the 'nic_name' value of each contained item.  Consequently, the
    'nic_name' key is required for each contained item, and it's value
    must be hashable. Updates to the 'mac' value of each item is handled
    specially, in that storage of incomplete byte lists, and/or values
    that do not pass the container class's mac_is_valid method,
    will not be stored.

New Class: VMNet_Style
    Make decisions about needed info from vm_type and driver_type params.
    The VMNet_Style_Map class variable is utilized for initial setup of
    VMNet derivatives (see below).  Essentially it helps provide a
    container-class type based on the contents of Params dict-like
    values for vm_type and driver_type.  It also establishes a mac
    address prefix to be used when generating MAC addresses.  However,
    any number of additional VMNet instance variables may also be set
    with default values.  The special 'default' keys are utilized when
    vm_type and/or driver_type don't match from Params.

New Class: ParamsNet
    VMNet sub-class specialized for initialization from a Params
    dict-like instance.  Essentially it filters out and initializes
    all networking info. for a particular VM.  Once initialized,
    networking information is accessed by NIC index or it's name (e.g.
    'nic1', 'nic2', etc.) in a dictionary-like or object property-like
    manner.  For example, instance[0].mac, instance('nic1').mac,
    instance['nic1']['mac'], etc.  Each nic object will be stored in
    the proper container class as determined by the VMNet_Style class.
    A utility generator method is provided for iteration over all
    VM's and NIC's mac address params definitions.  This makes validating
    or checking for mac address conflicts easy.

New Class DbNet:
    VMNet sub-class specialized for initialization from a database.
    Essentially analogous to ParamsNet, with a set of database
    management and locking methods.  Database items are hashed by the
    key provided at initialization time, so hash-clashes must be managed
    by implementation code.  Changes to any indexed NIC object
    automatically result in atomic database updates.  This also follows
    for any re-ordering or deletion of contained items.  The lock/unlock
    methods must be called externally prior to any database activity.
    Also provides a utility generator function for iteration over all
    MAC addresses contained within the database.

New Class: VirtNet (Re-write former networking code)
    Multi-inheritance of the ParamsNet and DbNet classes providing
    mediated networking info. management behavior combination of both.
    Provides the former generate_mac_address(), free_mac_address,
    set_mac_address(), and get_mac_address() functions, which behave
    exactly as before.  The only exception is the 'instance' parameter
    is no-longer required as it is established at initialization time.
    Setting and Freeing of arbitrary mac addresses is allowed (as
    before), however generation of new mac addresses is a sieve over
    cached values.  Deleting the database cache or freeing individual
    NICs MAC addresses will extend the pool within the confines of any
    mac prefixes defined by the responsible VMNet_Style definition.  A
    utility generator is provided which wrapps the mac_index generators
    from ParamsNet and DbNet (in that order).  This permits iteration
    over all known MAC address values possible.

+ Libvirt change details:
    Similar to KVM (below), however activate/deactivate_nic() are not
        yet implimented - no change from previous, unsupported
        hotplugging
    KVM/Xen guests make use of specialized MAC address prefixes
        + vm_type/driver_type combination chooses networking parameters
        + vm_type/driver_type combination dictates mac address prefix

+ KVM change details:

    Intended CodePath:

        Base Class __init__:
                Load/Parse Cartesian params + address_pool cache

        Base Class nic addition:
            BaseVM.add_nic
            BaseVM.activate_nic
            <testing>
            BaseVM.deactivate_nic
            BaseVM.del_nic

        KVM VM class __init__:
            Call BaseVM.__init__

        KVM VM create:
            vm.create:
                vm.add_nic -> vm.add_netdev
                specialized nic & netdev activation
            <testing>

        KVM VM nic addition:
            vm.add_netdev -or- vm.add_nic -> vm.add_netdev
                Uses BaseVM.add_nic
            vm.activate_netdev
            vm.activate_nic
            <testing>
            vm.deactivate_netdev
            vm.del_netdev
            vm.deactivate_nic
            vm.del_nic

   Modules Changes summary:

    BaseVM module
        Before:
            Different behavior per sub-class

        After:
            Defined new VMUnknownNetTypeError, raised when validating 'nettype'
            parameter doesn't match supported type.  Supported types left up
            to sub-classes to define.

    BaseVM.init
        Before:
            Subclasses and methods deal with params and cache DB ad hoc
        After:
            Common setup/validate networking from parameters and cache DB input

    BaseVM._generate_unique_id
        Before:
            vm.instance unique across thousands of VM instances created per
            second
        After:
            vm.instance unqie across millions of VM instances created per
            second

    BaseVM.get_mac_address
        Before:
            Obtain mac defined in params or cache DB input.  Raise
            VMMACAddressMissingError if no mac defined (note: doesn't actually
            check vm object in any way).
        After:
            Loading mac from params and/or cache db moved to __init__.  Check
            instance variables for mac and return or raise
            VMMACAddressMissingError

    BaseVM.get_address
        Before:
            Require sub-classes to impliment.
        After:
            Rely on __init__ to parse options and cache DB. If nic has mac
            defined, and corresponding IP in IP/MAC cache, verify in host
            arp cache then return IP.  Return localhost if no
            mac address defined for nic.

    BaseVM.free_mac_address
        Before:
            Different behavior per sub-class
        After:
            Unified behavior to drop mac reference only if they're not
            staticly defined.  When static, the original mac from params
            is re-established for the nic.

    BaseVM.add_nic
        Before:
            Left up to subclasses
        After:
            Adds new or updates existing NIC definition on vm instance with
        new or changed options.  Generates random nic_name if none specified.
        If IP and MAC are defined, updates IP/MAC cache.  Otherwise, generates
        random mac address.  Returns standardized instance of dict-like object
        containing NIC new/existing (possibly changed) parameters.

    BaseVM.del_nic
        Before:
            Left up to subclasses
        After:
            Drop references to mac in cache DB.  Drop references to IP
        from IP/MAC cache.  Ensure NIC parameters removed from vm instance.

    BaseVM.activate_nic
        Before:
            Host resource allocation bundled with sub-class add_nic method.
        After:
            Require sub-classes to implement separate from add_nic.

    BaseVM.deactivate_nic
        Before:
            Host resource deallocation bundled with sub-class add_nic method.
        After:
            Require sub-classes to implement separate from del_nic

    kvm_vm.init
        Before: Track nic-indexed netdev_id, device_id, and tapfd separately.
        After: Moved into BaseVM.__init__, single virt_utils.virtnet class.

    kvm_vm.clone
        unchanged- rely on __init__ to produce new virtnet instance

    kvm_vm.__make_qemu_command
        add_nic: unchanged
        add_net: unchanged
        Body: Referenced names, and assignment-order changes / updates
              so may be called independently of vm.create().

    kvm_vm.create
        Before:
            For each nic- Parse params, generating random netdev_id, device_id.
            For tap a.k.a. bridge devices, obtain ifname, bridge name, tapfd,
                add to bridge, then raise interface
            For each nic- obtain mac or generate random mac
                if params define ip address,update address cache
            __make_qemu_command
            background-run qemu command
            for each open tapfd- close tapfd
        After:
            (Rely on __init__ to parse params and cache db content)
            For each nic, use common add_nic(), but specialize activation:
                if nettype is bridge/tap:
                    open tap device node
                    register tapfd and tapfd_id to qemu
                    add interface to bridge
                    raise interface
                if nettype is user:
                    log info
            __make_qemu_command
            background-run qemu command
            for each already parsed nic:
                if tapfd open, close tapfd

    kvm_vm.destroy
        Same behavior, updated names

    kvm_vm.get_port
        Same behavior, updated names

    kvm_vm.get_peer
        Unchanged

    kvm_vm.get_address
        Before:
            Parses params for IP, if bridge/TAP mode defined for nic, parses
            params for mac address. Consults IP/MAC cache DB for IP.  Returns
            localhost if no IP defined or nic_mode is not 'tap'.
        After:
            Functionality made common by implimentation in BaseVM class.

    kvm_vm.get_ifname
        Before:
            Return the letter 't' followed by the vm's nic index, followed
            by a '-' and the last 11 characters of vm.instance.
        After:
            Retrieve ifname (generated in vm.add_netdev) for NIC or raise
            KeyError.

    kvm_vm.get_mac_address
        Before:
            Duplicated code from old BaseVM method with same name
        After:
            Functionality made common by implimentation in BaseVM class.

    kvm_vm.free_mac_address
        Before:
            Use virt_utils function to drop mac reference from cache DB.
        After:
            Functionality made common by implementation in BaseVM class.

    kvm_vm.add_netdev
        Before:
            Parse params for bridge name, generate random netdev_id, set vlan
            to number of other tap NICs on VM.  Generate ifname, open tap
            device node, set tapfd, bring up interface, generate random
            tapfd_id, run 'netdev_add' monitor command.  Verify netdev_id shows
            up in network info.  Return netdev_id.
        After:
            Rely on BaseVM.__init__ to parse options and cache DB for bridge
            name. Validate supported nettype specified in parameters ('bridge'
            or 'user') Generate random netdev_id, use defined vlan or set based
            on # total of nics on VM., and generate ifname.  Open tap device
            node moved to vm.activate_netdev().  Generate random tapfd_id.
            Running 'netdev_add' moved to vm.activate_netdev().  Return
            netdev_id.  Moved running of 'netdev_add' functionality into
            vm.activate_netdev.

    kvm_vm.del_netdev
        Before:
            Run 'netdev_del' monitor command.  Validate netdev_id no longer
            shows in network info.
        After:
            Moved 'netdev_del' monitor command and validation to
            vm.deactivate_nic().  Remove properties from nic: netdev_id,
            vlan, ifname, tapfd, tapfd_id.

    kvm_vm.add_nic
        Before:
            Take fixed set of nic parameters, generate a name for nic
            (nic_id) if not specified.  Call add_netdev() if netdev_id parameter
            not specified.  Generate random mac address if mac parameter not
            specified.  Setup/Run device_add monitor command with id=nic_id.
            Return dict containing nic parameters.  Verify nic_id shows up
            in qtree.
        After:
            Take arbitrary set of key/value nic prameters.  Call BaseVM.add_nic
            method with nic parameters.  Generate random device_id.  Call
            add_netdev with parameters if no netdev_id specified. Return
            standardized dict-like object containing nic parameters. Moved
            call to device_add command into vm.activate_nic method.

    kvm_vm.del_nic
        Before:
            Runs device_del monitor command.  Checks for wait timeout for
            nic to not show in qtree.
        After:
            Call BaseVM's del_nic method.  Functionality to
            call device_del moved to vm.deactivate_nic

    kvm_vm.activate_netdev
        Before:
            Functionality combined with vm.add_netdev
        After:
            Re-validate nettype is supported, setup netdev_add monitor
            command with parameters from nic based on nettype value. Run
            netdev_add monitor command.  Verify netdev_id shows up in network
            info.

    kvm_vm.deactivate_netdev
        Before:
            Functionality combined with vm.del_netdev
        After:
            Call netdev_del monitor command, verify netdev_id no longer shows
            in network info.

    kvm_vm.activate_nic
        Before:
            Functionality combined into vm.add_nic
        After:
            Setup/run device_add monitor command with id=nic_name.  Verify
            nic_name shows up in qtree.

    kvm_vm.deactivate_nic
        Before:
            Functionality combined with vm.del_nic
        After:
            Run 'device_del' monitor command. Validate netdev_id unknown to
            monitor. Raise virt_vm.VMDelNicError if qtree shows nic_name after
            wait time.

    kvm_vm.migrate
        Unchanged

Signed-off-by: Chris Evich <cevich@redhat.com>