Dynamic Multipoint VPN (DMVPN)

https://alpinelinux.org/about under Why the Name Alpine? states:
The first open-source implementation of Cisco's DMVPN, called OpenNHRP, was written for Alpine Linux.
So the aim of this document is to be the reference Linux DMVPN setup, with all the networking services needed for the clients that will use the DMVPN (DNS, firewall, etc.). Small Office Services offers additional services such as DHCP for clients, http proxying, and a basic SIP telephone system.
= Terminology =
;NBMA: Non-Broadcast Multi-Access network as described in RFC 2332
;Hub: the Next Hop Server (NHS) performing the Next Hop Resolution Protocol service within the NBMA cloud.
;Spoke: the Next Hop Resolution Protocol Client (NHC) which initiates NHRP requests of various types in order to obtain access to the NHRP service.
= Hardware =
If you need gigabits throughput you should go for a processor with AES-NI and SHA Extensions.
= Extract Certificates =
We will use certificates for DMVPN and for OpenVPN (RoadWarrior clients). If you are in need to generate your own certificates, please see Generating_SSL_certs_with_ACF. You should use a separate machine for this purpose. If you downloaded the certificates on a Windows machine, you may use WinSCP to copy them on the DMVPN box.
Here are the general purpose instruction for extracting certificates from pfx files:
Set appropriate permission for your certificate files:
= Spoke Node =
A local spoke node network has support for multiple ISP connections, along with redundant layer 2 switches. At least one 802.1q capable switch is required, and a second is optional for redundancy purposes. The typical spoke node network looks like:
== Boot Alpine USB == 
Follow the instructions on Create_a_Bootable_USB about how to create a bootable USB.
== Alpine Setup ==
We will setup the network interfaces as follows:
Interface 	Description 	Subnet
bond0.3 	Management 	10.1.0.129/26
bond0.101 	LAN 	10.1.0.0/25
bond0.256 	Internet from ISP1 	Allocated from ISP
bond0.257 	Internet from ISP2 	Allocated from ISP
bond0.620 	Transit between wifi proxy and dmvpn spoke node	10.1.0.252/30
bond0.701 	WiFi clients (no access to DMVPN network) 	172.17.48.0/24
bond0.1101	Voice 	10.2.0.0/24
You will be prompted something like this... 	Suggestion on what you could enter...
Select keyboard layout [none]: 	Type an appropriate layout for you
Select variant: 	Type an appropriate layout for you (if prompted)
Enter system hostname (short form, e.g. 'foo') [localhost]: 	Enter the hostname, e.g. vpnc
Available interfaces are: eth0Enter '?' for help on bridges, bonding and vlans.Which one do you want to initialize? (or '?' done') 	Enter bond0.101
Available bond slaves are: eth0 eth1Which slave(s) do you want to add to bond0? (or 'done') [eth0] 	eth0 eth1
IP address for bond0? (or 'dhcp', 'none', '?') [none]: 	Press Enter confirming 'none'
IP address for bond0.101? (or 'dhcp', 'none', '?') [dhcp]: 	Enter the IP address of your LAN interface, e.g. 10.1.0.1
Netmask? [255.255.255.0]: 	Press Enter confirming '255.255.255.0' or type another appropriate subnet mask
Gateway? (or 'none') [none]: 	Press Enter confirming 'none'
Do you want to do any manual network configuration? [no] 	yes
Make a copy of the bond0.101 configuration for bond0.620, bond0.701, bond0.1101, bond0.256 and bond0.257 (optional) interfaces.Don't forget to add a gateway and a metric value for ISP interfaces when multiple gateways are set.Save and close the file (:wq)	
DNS domain name? (e.g. 'bar.com') []: 	Enter the domain name of your intranet, e.g., example.net
DNS nameservers(s)? []: 	8.8.8.8 8.8.4.4 (we will change them later)
Changing password for rootNew password: 	Enter a secure password for the console
Retype password: 	Retype the above password
Which timezone are you in? ('?' for list) [UTC]: 	Press Enter confirming 'UTC'
HTTP/FTP proxy URL? (e.g. 'http://proxy:8080', or 'none') [none] 	Press Enter confirming 'none'
Enter mirror number (1-9) or URL to add (or r/f/e/done) [f]: 	Select a mirror close to you and press Enter
Which SSH server? ('openssh', 'dropbear' or 'none') [openssh]: 	Press Enter confirming 'openssh'
Which NTP client to run? ('openntpd', 'chrony' or 'none') [chrony]: 	Press Enter confirming 'chrony'
Which disk(s) would you like to use? (or '?' for help or 'none') [none]: 	Press Enter confirming 'none' or type 'none' if needed
Enter where to store configs ('floppy', 'usb' or 'none') [usb]: 	Press Enter confirming 'usb'
Enter apk cache directory (or '?' or 'none') [/media/usb/cache]: 	Press Enter confirming '/media/usb/cache'
== Networking ==
Update the networking configuration.
With your favorite editor open /etc/network/interfaces and add interfaces:
== Bonding ==
Update the bonding configuration.
With your favorite editor open /etc/network/interfaces and add bond-mode, bond-miimon and bond-updelay parameters to the bond0 stanza:
Bring up the new bonding settings:
== Physically install ==
At this point, you're ready to connect the VPN Spoke Node to the network if you haven't already done so. Please set up an 802.1q capable switch with the VLANs listed in AlpineSetup section. Once done, tag all of the VLANs on one port. Connect that port to eth0. Then, connect your first ISP's CPE to a switchport with VLAN 256 untagged.
== SSH ==
Remove password authentication and DNS reverse lookup:
Restart ssh:
== NTP server ==
In order to have attached devices syncing their time agains this host, we need to do some modifications to chrony config.<BR>
Add 'allow all' to the end of the '/etc/chrony/chrony.conf' so the file looks something like this:
Restart chronyd for the changes to take effect
 
== Recursive DNS ==
Install package(s):
With your favorite editor open /etc/unbound/unbound.conf and add the following configuration. If you have a domain that you want unbound to resolve but is internal to your network only, the stub-zone stanza is present:
Start unbound and start using unbound on this host:
== GRE Tunnel ==
With your favorite editor open /etc/network/interfaces and add the following:
Bring up the new gre1 interface:
== IPSEC ==
Install package(s):
With your favorite editor create /etc/ipsec.conf and set the content to the following:
Create missing directory:
Extract your pfx into /etc/racoon, using the filenames ca.pem, cert.pem, and key.pem (see instructions above for command).
With your favorite editor create /etc/racoon/racoon.conf and set the content to the following:
Edit /etc/conf.d/racoon and unset RACOON_PSK_FILE:
Start service(s):
== Next Hop Resolution Protocol (NHRP) ==
Install package(s):
With your favorite editor open /etc/opennhrp/opennhrp.conf and change the content to the following:
You must have a DNS A record hub.example.com for each hub node IP address.
With your favorite editor open /etc/opennhrp/opennhrp-script and change the content to the following:
Make it executable and start service(s):
== BGP ==
Install package(s):
With your favorite editor open /etc/quagga/bgpd.conf and change the content to the following (replace strongpassword with a password of your choice and %HUB_GRE_IP% with the Hub node GRE IP address):
* Add the line neighbor %HUB_GRE_IP% remote-as 65000 for each Hub host you have in your NBMA cloud.
Start service(s):
== OpenVPN ==
Install package(s):
Configure openvpn:
Start service(s):
== Firewall ==
Install package(s):
Enable IP forwarding:
With your favorite editor, edit the following files and set their contents as follows:
Activate the firewall:
== ISP Failover ==
Install package(s):
Configure pingu to monitor our bond0.256 and bond0.257 interfaces in /etc/pingu/pingu.conf. Add the hosts to monitor for ISP failover to /etc/pingu/pingu.conf and bind to primary ISP. We also set the ping timeout to 4 seconds.:
Make sure we can reach the public IP from our LAN by adding static route rules for our private net(s). Edit /etc/pingu/route-rules:
Start service(s):
Now, if both hosts stop responding to pings, ISP-1 will be considered down and all gateways via bond0.256 will be removed from main route table. Note that the gateway will not be removed from the route table '1'. This is so we can continue try ping via bond0.256 so we can detect that the ISP is back online. When ISP starts working again, the gateways will be added back to main route table again.
== Commit Configuration ==
Commit configuration:
= Hub Node =
We will document only what changes from the Spoke node setup.
== Routing Tables ==
echo -e "42\tnhrp_shortcut\n43\tnhrp_mtu\n44\tquagga\n>> /etc/iproute2/rt_tables
Add the following "up" commands:
== NHRP ==
With your favorite editor open /etc/opennhrp/opennhrp.conf on Hub 2 and set the content as follows:
Do the same on Hub 1 adding the data relative to Hub 2.
With your favorite editor open /etc/opennhrp/opennhrp-script and set the content as follows:
#!/bin/sh
case $1 in
interface-up)
	ip route flush proto 42 dev $NHRP_INTERFACE
	ip neigh flush dev $NHRP_INTERFACE
	;;
peer-register)
	CERT=`racoonctl get-cert inet $NHRP_SRCNBMA $NHRP_DESTNBMA | openssl x509 -inform der -text -noout | egrep -o "/OU=[^/]*(/[0-9]+)?" | cut -b 5-`
	if [ -z "`echo "$CERT" | grep "^GRE=$NHRP_DESTADDR"`" ]; then
		logger -t opennhrp-script -p auth.err "GRE registration of $NHRP_DESTADDR to $NHRP_DESTNBMA DENIED"
		exit 1
	fi
	logger -t opennhrp-script -p auth.info "GRE registration of $NHRP_DESTADDR to $NHRP_DESTNBMA authenticated"
	(
		flock -x 200
		
		AS=`echo "$CERT" | grep "^AS=" | cut -b 4-`
		vtysh -d bgpd -c "configure terminal" \
			-c "router bgp 65000" \
			-c "neighbor $NHRP_DESTADDR remote-as $AS" \
			-c "neighbor $NHRP_DESTADDR peer-group leaf" \
			-c "neighbor $NHRP_DESTADDR prefix-list net-$AS-in in"
		SEQ=5
		(echo "$CERT" | grep "^NET=" | cut -b 5-) | while read NET; do
			vtysh -d bgpd -c "configure terminal" \
				-c "ip prefix-list net-$AS-in seq $SEQ permit $NET le 26"
			SEQ=$(($SEQ+5))
		done
	) 200>/var/lock/opennhrp-script.lock
	;;
peer-up)
	echo "Create link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)"
	racoonctl establish-sa -w isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA || exit 1
	racoonctl establish-sa -w esp inet $NHRP_SRCNBMA $NHRP_DESTNBMA gre || exit 1 
	CERT=`racoonctl get-cert inet $NHRP_SRCNBMA $NHRP_DESTNBMA | openssl x509 -inform der -text -noout | egrep -o "/OU=[^/]*(/[0-9]+)?" | cut -b 5-`
	if [ -z "`echo "$CERT" | grep "^GRE=$NHRP_DESTADDR"`" ]; then
		logger -p daemon.err "GRE mapping of $NHRP_DESTADDR to $NHRP_DESTNBMA DENIED"
		exit 1
	fi
	if [ -n "$NHRP_DESTMTU" ]; then
		ARGS=`ip route get $NHRP_DESTNBMA from $NHRP_SRCNBMA | head -1`
		ip route add $ARGS proto 42 mtu $NHRP_DESTMTU table nhrp_mtu
	fi
	;;
peer-down)
	echo "Delete link from $NHRP_SRCADDR ($NHRP_SRCNBMA) to $NHRP_DESTADDR ($NHRP_DESTNBMA)"
	if [ "$NHRP_PEER_DOWN_REASON" != "lower-down" ]; then
		racoonctl delete-sa isakmp inet $NHRP_SRCNBMA $NHRP_DESTNBMA
	fi
	ip route del $NHRP_DESTNBMA src $NHRP_SRCNBMA proto 42 table nhrp_mtu
	;;
route-up)
	echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is up"
	ip route replace $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42 via $NHRP_NEXTHOP dev $NHRP_INTERFACE table nhrp_shortcut
	ip route flush cache
	;;
route-down)
	echo "Route $NHRP_DESTADDR/$NHRP_DESTPREFIX is down"
	ip route del $NHRP_DESTADDR/$NHRP_DESTPREFIX proto 42 table nhrp_shortcut
	ip route flush cache
	;;
esac
exit 0
== BGP ==
With your favorite editor open /etc/quagga/bgpd.conf on Hub 2 and set the content as follows:
Add the lines neighbor %Spoke1_GRE_IP%... for each spoke node you have. Do the same on Hub 1, changing the relevant data for Hub 2.
= Troubleshooting the DMVPN =
== Broken Path MTU Discovery (PMTUD) ==
ISPs afraid of ICMP (which is somehow legitimate) often just blindly add no ip unreachables in their router interfaces, effectively creating a blackhole router that breaks PMTUD, since ICMP Type 3 Code 4 packets (Fragmentation Needed) are dropped. PMTUD is needed by ISAKMP that runs on UDP (TCP works because it uses CLAMPMSS).
For technical details see https://packetlife.net/blog/2008/oct/9/disabling-unreachables-breaks-pmtud/
PMTUD could also be broken due to badly configured DSL modem/routers or bugged firmware. Turning off the firewall on modem itself or any VPN passthrough functionality it may help.
You can easily detect which host is the blackhole router by pinging with DF bit set and with packets of standard MTU size, each hop given in your traceroute to destination:
If you don't get a response back (either Echo-Response or Fragmentation-Needed) there's firewall dropping ICMP packets. If it answers to normal ping packets (DF bit cleared), most likely you have hit a blackhole router.
== Kernel and NHRP Routing Cache Issues ==
category: VPN