autotest/autotest — pull request #329 by cevich (2012-04-27 14:07:08+00:00)
Title: virt: Rewrite network/nic management

+ Integrated new network code with kvm module
+ Added many logging and debugging statements for new code
+ unit tests: New virt_utils_unittests tests
+ Updated shared ping test module
+ Updated kvm nic_bonding test module
+ Updated kvm nic_hotplug test module comment
+ virt_utils Env init now always warns when creating an empty env file
+ virt_utils Env init prints warning traceback when loading env fails
+ Pickling support for all new classes
+ Raise exception when adding nic with clashing name
+ New virt_utils_unittest deletes temporary testing database
+ IP/MAC cache is only updated with new entries
+ VM cleanup resets static addresses properly
+ Enhanced output of needs_restart() VM comparison function
+ Enhanced Env class to output more details on non-fatal load failures
+ Old virt_util functions all raise depreciation warnings
to aide bug squashing
+ remove unnecessary nic_mode from subtests parameter, rely on default
nettype=bridge instead
+ Implimented new nic/net management within libvirt_vm
+ Added stubbs for nic activation/deactivation in libvirt_vm
+ Removed extranious netdev_id/device_id/tap/ifname support from libvirt_vm
+ Enhanced logging for vm.needs_restart()
+ Changed libvirt vm to only free mac addresses for undefined VMs
+ Improved libvirt vm.is_persistent() to handle non-existing VMs
+ Defined mac address prefix convensions for libvirt (kvm/xen)
+ virt config: New universal virt. networking configuration
nettype + netdst parameters determine networking style and
destination for virtual NICs. Separate NIC sets can be defined
for each VM. Documented examples of other per-nic parameters
in config. comments.
+ virt_utils change etails:

New Class: PropCan
Allow container-like access to fixed set of items (__slots__)
This allows NIC/Net handling code to lock-down a fixed set of
parameters relevant to a vm_type / driver_type combination.
Data is accessible in a dictionary-like manner or as object
properties, however empty or None value items will pretend to
not exist. The special empty/None handling optimises access
and storage by only representing values containing meaningful
data.

New Class: Base_VirtIface
Networking information for single guest interface and host
connection. This is an abstract base-child-class of PropCan
containing the minimal amount of networking keys needed, along
with some basic name and mac address handling methods. Some
amount of effort is made to ensure stored mac address data
is in an expected format - two ASCII character hex-byte
representation, separated by :'s, six or less bytes in total.
Complete mac addresses are generated by filling in missing
trailing bytes from a sequential pool maintained as a class
variable. Numbering starts randomly from 0-255, then proceeds
incrementally for each byte requested. Sequential byte generation
used to ease unit testing.

New Class: Libvirt_VirtIface & KVM_VirtIface
Specializations of Base_VirtIface parent class (and PropCan)
with networking keys specific to Libvirt or KVM respectively.
If a new type or variation is needed, simply subclass Base_VirtIface
and update the VMNet_Style.VMNet_Style_Map as needed. The new or
variant class will be used automatically based on resolving
vm_type, and/or driver_type from the supplied Params dict (see
other classes below)

New Class: VMNet
Abstract base-class collection of networking information, stored
within a particular container class (e.g. a subclass of
Base_VirtIface). This class behaves like a list of PropCans.
It may be initially empty or created from a list-like container
of dictionary-like objects holding key/value pairs of networking
data. Items within a VMNet instance are indexed by number and
the 'nic_name' value of each contained item. Consequently, the
'nic_name' key is required for each contained item, and it's value
must be hashable. Updates to the 'mac' value of each item is handled
specially, in that storage of incomplete byte lists, and/or values
that do not pass the container class's mac_is_valid method,
will not be stored.

New Class: VMNet_Style
Make decisions about needed info from vm_type and driver_type params.
The VMNet_Style_Map class variable is utilized for initial setup of
VMNet derivatives (see below). Essentially it helps provide a
container-class type based on the contents of Params dict-like
values for vm_type and driver_type. It also establishes a mac
address prefix to be used when generating MAC addresses. However,
any number of additional VMNet instance variables may also be set
with default values. The special 'default' keys are utilized when
vm_type and/or driver_type don't match from Params.

New Class: ParamsNet
VMNet sub-class specialized for initialization from a Params
dict-like instance. Essentially it filters out and initializes
all networking info. for a particular VM. Once initialized,
networking information is accessed by NIC index or it's name (e.g.
'nic1', 'nic2', etc.) in a dictionary-like or object property-like
manner. For example, instance[0].mac, instance('nic1').mac,
instance['nic1']['mac'], etc. Each nic object will be stored in
the proper container class as determined by the VMNet_Style class.
A utility generator method is provided for iteration over all
VM's and NIC's mac address params definitions. This makes validating
or checking for mac address conflicts easy.

New Class DbNet:
VMNet sub-class specialized for initialization from a database.
Essentially analogous to ParamsNet, with a set of database
management and locking methods. Database items are hashed by the
key provided at initialization time, so hash-clashes must be managed
by implementation code. Changes to any indexed NIC object
automatically result in atomic database updates. This also follows
for any re-ordering or deletion of contained items. The lock/unlock
methods must be called externally prior to any database activity.
Also provides a utility generator function for iteration over all
MAC addresses contained within the database.

New Class: VirtNet (Re-write former networking code)
Multi-inheritance of the ParamsNet and DbNet classes providing
mediated networking info. management behavior combination of both.
Provides the former generate_mac_address(), free_mac_address,
set_mac_address(), and get_mac_address() functions, which behave
exactly as before. The only exception is the 'instance' parameter
is no-longer required as it is established at initialization time.
Setting and Freeing of arbitrary mac addresses is allowed (as
before), however generation of new mac addresses is a sieve over
cached values. Deleting the database cache or freeing individual
NICs MAC addresses will extend the pool within the confines of any
mac prefixes defined by the responsible VMNet_Style definition. A
utility generator is provided which wrapps the mac_index generators
from ParamsNet and DbNet (in that order). This permits iteration
over all known MAC address values possible.

+ Libvirt change details:
Similar to KVM (below), however activate/deactivate_nic() are not
yet implimented - no change from previous, unsupported
hotplugging
KVM/Xen guests make use of specialized MAC address prefixes
+ vm_type/driver_type combination chooses networking parameters
+ vm_type/driver_type combination dictates mac address prefix

+ KVM change details:

Intended CodePath:

Base Class __init__:
Load/Parse Cartesian params + address_pool cache

Base Class nic addition:
BaseVM.add_nic
BaseVM.activate_nic

BaseVM.deactivate_nic
BaseVM.del_nic

KVM VM class __init__:
Call BaseVM.__init__

KVM VM create:
vm.create:
vm.add_nic -> vm.add_netdev
specialized nic & netdev activation

KVM VM nic addition:
vm.add_netdev -or- vm.add_nic -> vm.add_netdev
Uses BaseVM.add_nic
vm.activate_netdev
vm.activate_nic

vm.deactivate_netdev
vm.del_netdev
vm.deactivate_nic
vm.del_nic

Modules Changes summary:

BaseVM module
Before:
Different behavior per sub-class

After:
Defined new VMUnknownNetTypeError, raised when validating 'nettype'
parameter doesn't match supported type. Supported types left up
to sub-classes to define.

BaseVM.init
Before:
Subclasses and methods deal with params and cache DB ad hoc
After:
Common setup/validate networking from parameters and cache DB input

BaseVM._generate_unique_id
Before:
vm.instance unique across thousands of VM instances created per
second
After:
vm.instance unqie across millions of VM instances created per
second

BaseVM.get_mac_address
Before:
Obtain mac defined in params or cache DB input. Raise
VMMACAddressMissingError if no mac defined (note: doesn't actually
check vm object in any way).
After:
Loading mac from params and/or cache db moved to __init__. Check
instance variables for mac and return or raise
VMMACAddressMissingError

BaseVM.get_address
Before:
Require sub-classes to impliment.
After:
Rely on __init__ to parse options and cache DB. If nic has mac
defined, and corresponding IP in IP/MAC cache, verify in host
arp cache then return IP. Return localhost if no
mac address defined for nic.

BaseVM.free_mac_address
Before:
Different behavior per sub-class
After:
Unified behavior to drop mac reference only if they're not
staticly defined. When static, the original mac from params
is re-established for the nic.

BaseVM.add_nic
Before:
Left up to subclasses
After:
Adds new or updates existing NIC definition on vm instance with
new or changed options. Generates random nic_name if none specified.
If IP and MAC are defined, updates IP/MAC cache. Otherwise, generates
random mac address. Returns standardized instance of dict-like object
containing NIC new/existing (possibly changed) parameters.

BaseVM.del_nic
Before:
Left up to subclasses
After:
Drop references to mac in cache DB. Drop references to IP
from IP/MAC cache. Ensure NIC parameters removed from vm instance.

BaseVM.activate_nic
Before:
Host resource allocation bundled with sub-class add_nic method.
After:
Require sub-classes to implement separate from add_nic.

BaseVM.deactivate_nic
Before:
Host resource deallocation bundled with sub-class add_nic method.
After:
Require sub-classes to implement separate from del_nic

kvm_vm.init
Before: Track nic-indexed netdev_id, device_id, and tapfd separately.
After: Moved into BaseVM.__init__, single virt_utils.virtnet class.

kvm_vm.clone
unchanged- rely on __init__ to produce new virtnet instance

kvm_vm.__make_qemu_command
add_nic: unchanged
add_net: unchanged
Body: Referenced names, and assignment-order changes / updates
so may be called independently of vm.create().

kvm_vm.create
Before:
For each nic- Parse params, generating random netdev_id, device_id.
For tap a.k.a. bridge devices, obtain ifname, bridge name, tapfd,
add to bridge, then raise interface
For each nic- obtain mac or generate random mac
if params define ip address,update address cache
__make_qemu_command
background-run qemu command
for each open tapfd- close tapfd
After:
(Rely on __init__ to parse params and cache db content)
For each nic, use common add_nic(), but specialize activation:
if nettype is bridge/tap:
open tap device node
register tapfd and tapfd_id to qemu
add interface to bridge
raise interface
if nettype is user:
log info
__make_qemu_command
background-run qemu command
for each already parsed nic:
if tapfd open, close tapfd

kvm_vm.destroy
Same behavior, updated names

kvm_vm.get_port
Same behavior, updated names

kvm_vm.get_peer
Unchanged

kvm_vm.get_address
Before:
Parses params for IP, if bridge/TAP mode defined for nic, parses
params for mac address. Consults IP/MAC cache DB for IP. Returns
localhost if no IP defined or nic_mode is not 'tap'.
After:
Functionality made common by implimentation in BaseVM class.

kvm_vm.get_ifname
Before:
Return the letter 't' followed by the vm's nic index, followed
by a '-' and the last 11 characters of vm.instance.
After:
Retrieve ifname (generated in vm.add_netdev) for NIC or raise
KeyError.

kvm_vm.get_mac_address
Before:
Duplicated code from old BaseVM method with same name
After:
Functionality made common by implimentation in BaseVM class.

kvm_vm.free_mac_address
Before:
Use virt_utils function to drop mac reference from cache DB.
After:
Functionality made common by implementation in BaseVM class.

kvm_vm.add_netdev
Before:
Parse params for bridge name, generate random netdev_id, set vlan
to number of other tap NICs on VM. Generate ifname, open tap
device node, set tapfd, bring up interface, generate random
tapfd_id, run 'netdev_add' monitor command. Verify netdev_id shows
up in network info. Return netdev_id.
After:
Rely on BaseVM.__init__ to parse options and cache DB for bridge
name. Validate supported nettype specified in parameters ('bridge'
or 'user') Generate random netdev_id, use defined vlan or set based
on # total of nics on VM., and generate ifname. Open tap device
node moved to vm.activate_netdev(). Generate random tapfd_id.
Running 'netdev_add' moved to vm.activate_netdev(). Return
netdev_id. Moved running of 'netdev_add' functionality into
vm.activate_netdev.

kvm_vm.del_netdev
Before:
Run 'netdev_del' monitor command. Validate netdev_id no longer
shows in network info.
After:
Moved 'netdev_del' monitor command and validation to
vm.deactivate_nic(). Remove properties from nic: netdev_id,
vlan, ifname, tapfd, tapfd_id.

kvm_vm.add_nic
Before:
Take fixed set of nic parameters, generate a name for nic
(nic_id) if not specified. Call add_netdev() if netdev_id parameter
not specified. Generate random mac address if mac parameter not
specified. Setup/Run device_add monitor command with id=nic_id.
Return dict containing nic parameters. Verify nic_id shows up
in qtree.
After:
Take arbitrary set of key/value nic prameters. Call BaseVM.add_nic
method with nic parameters. Generate random device_id. Call
add_netdev with parameters if no netdev_id specified. Return
standardized dict-like object containing nic parameters. Moved
call to device_add command into vm.activate_nic method.

kvm_vm.del_nic
Before:
Runs device_del monitor command. Checks for wait timeout for
nic to not show in qtree.
After:
Call BaseVM's del_nic method. Functionality to
call device_del moved to vm.deactivate_nic

kvm_vm.activate_netdev
Before:
Functionality combined with vm.add_netdev
After:
Re-validate nettype is supported, setup netdev_add monitor
command with parameters from nic based on nettype value. Run
netdev_add monitor command. Verify netdev_id shows up in network
info.

kvm_vm.deactivate_netdev
Before:
Functionality combined with vm.del_netdev
After:
Call netdev_del monitor command, verify netdev_id no longer shows
in network info.

kvm_vm.activate_nic
Before:
Functionality combined into vm.add_nic
After:
Setup/run device_add monitor command with id=nic_name. Verify
nic_name shows up in qtree.

kvm_vm.deactivate_nic
Before:
Functionality combined with vm.del_nic
After:
Run 'device_del' monitor command. Validate netdev_id unknown to
monitor. Raise virt_vm.VMDelNicError if qtree shows nic_name after
wait time.

kvm_vm.migrate
Unchanged

Signed-off-by: Chris Evich