XenDom0VLANstoDomUVirtualNICs

__NOTOC__
== Introduction ==
== Examples ==
These are very simple instructions for mapping VLANs on the Dom0 through to the DomU NICs. These are Redhat/CentOS specific, and in this example the NICs were mapped through to a HVM Windoze box.
Other descriptions to help being picked up by search engines is: 
# VLAN trunk to virtual machine mapping 
# VLANs to DomU interfaces
Basic Flow: 
# Create a bridge group for each .1q interface you want, and add the interfaces to the bridge. 
# Map each virtual NIC to the bridge group. 
# Don't forget to set the MTU in Windoze
# Configure the bond interface, and create VLAN interfaces on the bond interface. 
# Create a bridge and map each vlan interface into the bridge. 
All the files for this are at /etc/sysconfig/network-scripts/. (This examples creates VLAN 900-903.) You will need to change the HWADDR to match you NIC cards.
These are all the related files:
<nowiki>
/etc/sysconfig/network-scripts/ifcfg-eth0
/etc/sysconfig/network-scripts/ifcfg-eth1
/etc/sysconfig/network-scripts/ifcfg-bond0
/etc/sysconfig/network-scripts/ifcfg-bond0.900
/etc/sysconfig/network-scripts/ifcfg-bond0.901
/etc/sysconfig/network-scripts/ifcfg-bond0.902
/etc/sysconfig/network-scripts/ifcfg-bond0.903
/etc/sysconfig/network-scripts/ifcfg-br900
/etc/sysconfig/network-scripts/ifcfg-br901
/etc/sysconfig/network-scripts/ifcfg-br902
/etc/sysconfig/network-scripts/ifcfg-br903
</nowiki>
1.1 These files configure the real ethernet ports to be part of an active/standby 'bond' interface.
/etc/sysconfig/network-scripts/ifcfg-eth0
<nowiki>
# Intel Corporation 82541GI Gigabit Ethernet Controller
DEVICE=eth0
BOOTPROTO=none
HWADDR=00:15:17:28:f6:34
ONBOOT=yes
TYPE=Ethernet
#ETHTOOL_OPTS="autoneg off speed 100 duplex full"
MASTER=bond0
SLAVE=yes
USERCTL=no
</nowiki>
/etc/sysconfig/network-scripts/ifcfg-eth1
<nowiki>
# Intel Corporation 82566DM-2 Gigabit Network Connection
DEVICE=eth1
BOOTPROTO=none
HWADDR=00:15:17:28:f6:36
ONBOOT=yes
TYPE=Ethernet
#ETHTOOL_OPTS="autoneg off speed 100 duplex full"
MASTER=bond0
SLAVE=yes
USERCTL=no
</nowiki>
1.2 This is the bond interface configuration.
Here the bond0 interface, in Dom0, is actually getting a IP via dhcp. You may want to set BOOTPROTO=none if you don't want this. The 'virtual mac' address is set here. Make sure this is unique.
/etc/sysconfig/network-scripts/ifcfg-bond0
<nowiki>
DEVICE=bond0
BOOTPROTO=dhcp
ONBOOT=yes
USERCTL=no
#PEERDNS=no
IPV6INIT=no
MACADDR=52:54:00:12:45:56
BONDING_OPTS="mode=1 miimon=100 primary=eth0"
</nowiki>
1.3 Now create the bridges.
/etc/sysconfig/network-scripts/ifcfg-br900
<nowiki>
DEVICE=br900
TYPE=Bridge
#STP=on
ONBOOT=yes
BOOTPROTO=none
</nowiki>
...
<nowiki>
DEVICE=br903
TYPE=Bridge
#STP=on
ONBOOT=yes
BOOTPROTO=none
</nowiki>
1.4 Now the creation of all the VLAN interfaces, which are put into the bridges.
/etc/sysconfig/network-scripts/ifcfg-bond0.900
<nowiki>
DEVICE=bond0.900
PHYSDEV=bond0
ONBOOT=yes
BOOTPROTO=none
VLAN=yes
BRIDGE=br900
</nowiki>
...
<nowiki>
DEVICE=bond0.903
PHYSDEV=bond0
ONBOOT=yes
BOOTPROTO=none
VLAN=yes
BRIDGE=br903
</nowiki>
2. Load the bond-ing module.The main thing to add is below (rest of the bond is set in the file above). Reload or modprobe.
<nowiki>
alias bond0 bonding
</nowiki>
3. Finally, the important line in the Xen xm config file is:
<nowiki>
vif = [ 'type=ioemu, bridge=br900', 'type=ioemu, bridge=br901',
'type=ioemu, bridge=br902', 'type=ioemu, bridge=br903' ]
</nowiki>
4. Don't forget to set the MTU to 1494 using the 'regedit' if you are using windoze.
----
 If you are using Centos/Redhat you can also use 'virsh' which has a XML virtual machine definition. 'virsh' seems to be strongly supported by Redhat, and XML is fairly clean, so instead of just using the 'xm' style files do this:
<nowiki>
<domain type='xen'>
 <name>vm_name</name>
 <memory>256000</memory>
 <bootloader>/usr/bin/pygrub</bootloader>
 <on_poweroff>destroy</on_poweroff>
 <on_reboot>restart</on_reboot>
 <on_crash>restart</on_crash>
 <vcpu>2</vcpu>
 <devices>
 <disk type='file' device='disk'>
 <driver name='tap'/>
 <source file='/home/xen/my_vm_centos52.img'/>
 <target dev='xvda' bus='xen'/>
 </disk>
 <interface type='bridge'>
 <source bridge='br900'/>
 <mac address='00:16:2e:62:f0:1c'/>
 <script path='vif-bridge'/>
 </interface>
 <interface type='bridge'>
 <source bridge='br901'/>
 <mac address='00:16:2e:62:f1:1c'/>
 <script path='vif-bridge'/>
 </interface>
 <interface type='bridge'>
 <source bridge='br902'/>
 <mac address='00:16:2e:62:31:1c'/>
 <script path='vif-bridge'/>
 </interface>
 <interface type='bridge'>
 <source bridge='br903'/>
 <mac address='00:16:2e:62:31:3c'/>
 <script path='vif-bridge'/>
 </interface>
 </devices>
</domain>
</nowiki>
Do 'virsh create vm_name.xml', 'virsh list', and 'virsh autostart vm_name'
----
regards, Dave Seddon
Here are some additional notes on roughly the same implementation using QEMU/KVM. The key here is again, setting up the correct bridge groups and linking the virtual NICs to these.
----
These are some KVM/QEMU instructions for giving virtual machines
multiple NICs bound to multiple physical NICs.
Eg.
<nowiki>
Eth0 - Br0 - tap0 - Virtual NIC 0
Eth1 - Br1 - tap1 - Virtual NIC 1
</nowiki>
Linux host configuration (debian/ubuntu)
This setups up the two bridge interfaces (no STP)
<nowiki>
# The primary network interface
auto br0
iface br0 inet dhcp
 bridge_stp off
 bridge_ports eth0
iface eth0 inet static
address 0.0.0.0
netmask 0.0.0.0
auto eth0
iface eth1 inet static
 address 0.0.0.0
 netmask 0.0.0.0
auto eth1
auto br1
iface br1 inet static
 address 0.0.0.0
 netmask 0.0.0.0
 bridge_stp off
 bridge_ports eth1
</nowiki>
To avoid KVM having to create the tap devices when it starts up (which required root), setup an init script to generate the interfaces with the correct ownership.
You will need to change 'das' to whatever username you are using to run KVM. Recommend starting this script after the /etc/init.d/network runs.
<nowiki>
root@das-hp:/home/das/qemu# cat /etc/init.d/make_tap_devices.sh
#!/bin/sh
#make tap0
/usr/sbin/tunctl -u das
#make tap1
/usr/sbin/tunctl -u das
</nowiki>
KVM normally calls this script when creating tap interfaces:
(this script matches the bridge/switch on the default route - so usually
br0, for example)
<nowiki>
root@das-hp:/etc/kvm# cat kvm-ifup
#!/bin/sh
switch=$(ip route ls | awk '/^default / { for(i=0;i<NF;i++) { if ($(i)
== "dev") print $(i+1) }}')
/sbin/ifconfig $1 0.0.0.0 up
/usr/sbin/brctl addif ${switch} $1
exit 0
</nowiki>
However, we need a script that will match the first argument and change
this to the correct bridge/switch:
e.g. This script is called via kvm when creating a new tap interface
kvm-ifup-tap tap0 -> add to bridge br0
kvm-ifup-tap tap1 -> add to bridge br1
<nowiki>
root@das-hp:/etc/kvm# cat kvm-ifup-tap
#!/bin/sh
#this script is called via kvm when creating a new tap interface
# $1 = tap0, or tap1, or whatever tap device
bridge=$(echo $1 | sed -e 's/tap//')
#bring the tap interface up
/sbin/ifconfig $1 0.0.0.0 up
#add the tap interface to the bridge
/usr/sbin/brctl addif br${bridge} $1
exit 0
</nowiki>
Then finally the command to run all this is:
<nowiki>
sudo /usr/bin/kvm -m 1100 \
 -localtime \
 -usbdevice tablet \
 -net nic,vlan=0,model=e1000,macaddr=52:54:00:12:34:57 -net tap,vlan=0,ifname=tap0,script=/etc/kvm/kvm-ifup-tap \
 -net nic,vlan=1,model=e1000,macaddr=52:54:00:12:34:58 -net tap,vlan=1,ifname=tap1,script=/etc/kvm/kvm-ifup-tap \
 /home/das/qemu/winxp10g.img.qcow
</nowiki>
Notes:
1. Make sure the 'vlan=X' the Xes match. This is how the virtual nic is associated with the tap interface. The NIC does not generate tagged frames. If the second NIC line above also had vlan=0, then all the NICs are effectively part of the same VLAN (and causes loops)
2. The above command is staring windoze, however it will work for other OSes too (e.g. Juniper olives)
3. The MAC addresses are hard coded to make sure DHCP address assignment is consistent)
Alternatively, you can replace the existing kvm-ifup script (so you
don't need to specify this to kvm):
<nowiki>
mv kvm-ifup not.kvm-ifup
ln -s kvm-ifup-tap kvm-ifup
{{{
root@das-hp:/etc/kvm# ls -la
total 28
drwxr-xr-x 3 root root 4096 2009-04-16 11:20 .
drwxr-xr-x 174 root root 12288 2009-04-16 10:44 ..
lrwxrwxrwx 1 root root 12 2009-04-16 11:20 kvm-ifup -> kvm-ifup-tap
-rwxr-xr-x 1 root root 294 2009-04-16 11:19 kvm-ifup-tap
-rwxr-xr-x 1 root root 181 2008-04-23 04:35 not.kvm-ifup
drwxr-xr-x 2 root root 4096 2008-10-06 13:13 utils
</nowiki>
KVM command becomes:
<nowiki>
sudo /usr/bin/kvm -m 1100 \
 -localtime \
 -usbdevice tablet \
 -net nic,vlan=0,model=e1000,macaddr=52:54:00:12:34:57 -net tap,vlan=0,ifname=tap0 \
 -net nic,vlan=1,model=e1000,macaddr=52:54:00:12:34:58 -net tap,vlan=1,ifname=tap1 \
 /home/das/qemu/winxp10g.img.qcow
</nowiki>
Notes:
1. Make sure the 'vlan=X' the Xes match. This is how the virtual nic is associated with the tap interface. The NIC does not generate tagged frames. If the second NIC line above also had vlan=0, then all the NICs are effectively part of the same VLAN (and causes loops)
2. The above command is staring windoze, however it will work for other OSes too (e.g. Juniper olives)
3. The MAC addresses are hard coded to make sure DHCP address assignment is consistent)
Some other interesting output:
<nowiki>
root@das-hp:/etc/kvm# ifconfig -a
br0 Link encap:Ethernet HWaddr 00:1a:4b:5a:b4:96
 inet addr:XX.XX.XX.36 Bcast:XX.XX.XX.255 Mask:255.255.255.0
 inet6 addr: fe80::21a:4bff:fe5a:b496/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:6892185 errors:0 dropped:0 overruns:0 frame:0
 TX packets:682 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:896362303 (854.8 MB) TX bytes:274918 (268.4 KB)
br1 Link encap:Ethernet HWaddr 00:ff:eb:19:39:9b
 inet6 addr: fe80::1211:ff:fe00:477/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:576 errors:0 dropped:0 overruns:0 frame:0
 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:85748 (83.7 KB) TX bytes:468 (468.0 B)
eth0 Link encap:Ethernet HWaddr 00:1a:4b:5a:b4:96
 inet6 addr: fe80::21a:4bff:fe5a:b496/64 Scope:Link
 UP BROADCAST MULTICAST MTU:1500 Metric:1
 RX packets:3014 errors:0 dropped:0 overruns:0 frame:0
 TX packets:936 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:422764 (412.8 KB) TX bytes:219484 (214.3 KB)
 Interrupt:16
eth1 Link encap:Ethernet HWaddr 10:11:00:00:04:77
 inet6 addr: fe80::1211:ff:fe00:477/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:291 errors:3464688 dropped:0 overruns:0 frame:0
 TX packets:291 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:1000
 RX bytes:43297 (42.2 KB) TX bytes:47029 (45.9 KB)
lo Link encap:Local Loopback
 inet addr:127.0.0.1 Mask:255.0.0.0
 inet6 addr: ::1/128 Scope:Host
 UP LOOPBACK RUNNING MTU:16436 Metric:1
 RX packets:16002 errors:0 dropped:0 overruns:0 frame:0
 TX packets:16002 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:0
 RX bytes:12053229 (11.4 MB) TX bytes:12053229 (11.4 MB)
tap0 Link encap:Ethernet HWaddr 00:ff:62:c1:3b:12
 inet6 addr: fe80::2ff:62ff:fec1:3b12/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:174 errors:0 dropped:0 overruns:0 frame:0
 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:25728 (25.1 KB) TX bytes:468 (468.0 B)
tap1 Link encap:Ethernet HWaddr 00:ff:eb:19:39:9b
 inet6 addr: fe80::2ff:ebff:fe19:399b/64 Scope:Link
 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
 RX packets:90 errors:0 dropped:0 overruns:0 frame:0
 TX packets:90 errors:0 dropped:0 overruns:0 carrier:0
 collisions:0 txqueuelen:500
 RX bytes:13110 (12.8 KB) TX bytes:13086 (12.7 KB)
</nowiki>
<nowiki>
root@das-hp:/etc/kvm# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.001a4b5ab496 no eth0
 tap0
br1 8000.00ffeb19399b no eth1
 tap1
</nowiki>
Category:Xen
Category:Users
Category:Example
Category:Beginners
Category:HowTo