Portal:Cloud VPS/Admin/Neutron LVS

This page contains information on running LVS in our CloudVPS environment using Neutron.
Due to the nature and complexity of this use case, this is an experimental setup, and not a public end-user offered service.
= Experimental setup =
To test LVS-DR, ldirectord was used to manage ipvsadm configuration. I didn't check any actual ops/puppet.git LVS roles to be able to simplify the setup.
I didn't test with floating IPs as VIPs (public addresses) to simplify the setup.
* Create a Cloud VPS project lvs-test in the codfw1dev-r region
* Create 3 VM instances in the lvs-test project: lvs-server (172.16.128.214), lvs-backend-01 (172.16.128.212) and lvs-backend-02 (172.16.128.213).
* Create and assign an additional IP address for lvs-server: 172.16.128.211 (this will be our VIP). nova add-fixed-ip <server> <network-id>
* SSH to VMs and install basic packages: tcpdump, netcat, ipvsadm, ldirectord.
* Add VIP to the lo interfaces on backend servers ip addr add 172.16.128.211/32 dev lo and to eth0 in the LVS server.
* Configure ldirectord with a very basic config (in the lvs-server VM) /etc/ha.d/ldirectord.cf:
checktimeout=3
checkinterval=1
autoreload=yes
quiescent=no
virtual=172.16.128.211:80
 servicename=test
 comment=test
 real=172.16.128.212:80 gate
 real=172.16.128.213:80 gate
 service=http
 scheduler=rr
 protocol=tcp
 checktype=ping
* Check resulting ipvsadm configuration:
root@lvs-server:~# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
 -> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.128.211:80 rr
 -> 172.16.128.212:80 Route 1 0 0 
 -> 172.16.128.213:80 Route 1 0 0 
* Start testing, from the Neutron server, nothing works with LVS yet (here, the Neutron server acts as the external client):
aborrero@labtestneutron2001:~$ nc 172.16.128.211 80
^C
* Now (WARNING) beging disabling security features for the involved ports:
root@labtestcontrol2003:~# neutron port-list | egrep 211\|212\|214\|213
| 64d95341-9f1c-469b-a69a-da732b423c0d | | fa:16:3e:5e:9e:f0 | {"subnet_id": "7adfcebe-b3d0-4315-92fe-e8365cc80668", "ip_address": "172.16.128.213"} |
| 7306ee5f-42ca-43aa-a25a-9993c899c428 | | fa:16:3e:19:48:05 | {"subnet_id": "7adfcebe-b3d0-4315-92fe-e8365cc80668", "ip_address": "172.16.128.211"} |
| | | | {"subnet_id": "7adfcebe-b3d0-4315-92fe-e8365cc80668", "ip_address": "172.16.128.214"} |
| b166d7aa-3b8f-49f0-a220-b341bda10f4a | | fa:16:3e:1f:35:9a | {"subnet_id": "7adfcebe-b3d0-4315-92fe-e8365cc80668", "ip_address": "172.16.128.212"} |
* disable security group (iptables rules) and port-security (ebtables rules) for all instances Enable the VIP address to circulate in each VM ports:
user@cloudcontrol1004:~$ sudo wmcs-openstack port set --allowed-address ip-address=172.16.128.211 $PORT_UUID_VM1
[..]
user@cloudcontrol1004:~$ sudo wmcs-openstack port set --allowed-address ip-address=172.16.128.211 $PORT_UUID_VM2
[..]
* check if LVS works now:
aborrero@labtestneutron2001:~ 1 $ nc 172.16.128.211 80
(UNKNOWN) [172.16.128.211] 80 (http) : Connection refused
aborrero@labtestneutron2001:~ 1 $ nc 172.16.128.211 80
(UNKNOWN) [172.16.128.211] 80 (http) : Connection refused
root@lvs-server:~# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
 -> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 172.16.128.211:80 rr
 -> 172.16.128.212:80 Route 1 1 0 
 -> 172.16.128.213:80 Route 1 1 0
root@lvs-backend-01:~# tcpdump -n -i any tcp port 80
12:28:42.570302 IP 10.192.20.4.41844 > 172.16.128.211.80: Flags [S], seq 2177770907, win 29200, options [mss 1460,sackOK,TS val 2027567397 ecr 0,nop,wscale 9], length 0
12:28:42.570333 IP 172.16.128.211.80 > 10.192.20.4.41844: Flags [R.], seq 0, ack 2177770908, win 0, length 0
root@lvs-backend-02:~# tcpdump -n -i any tcp port 80
12:28:40.271707 IP 10.192.20.4.41842 > 172.16.128.211.80: Flags [S], seq 3771176807, win 29200, options [mss 1460,sackOK,TS val 2027567281 ecr 0,nop,wscale 9], length 0
12:28:40.271740 IP 172.16.128.211.80 > 10.192.20.4.41842: Flags [R.], seq 0, ack 3771176808, win 0, length 0
* it works!
= Conclusions =
* This setup is very similar to what we do with Portal:Cloud_VPS/Admin/Keepalived
* In the last few openstack versions this has been simplified a bit, but the horizon workflow is a still not very easy to handle, see Phabricator T255670 - horizon: enable neutron port management.