Multi ISP

This document describes how to configure your Alpine Linux router with multiple ISPs, with or without failover and load balancing.
Please note: Alpine Linux v2.4.4 or newer is required.
Interface	Name	IP address 	Description
eth0 	ISP1	DHCP 	Internet with static IP address (we will use 1.2.3.4/24 and gw 1.2.3.1 as example)
eth1 	DMZ 	192.168.0.1	Demilitarized zone
eth2 	LAN 	192.168.1.1	Local network
eth3 	ISP2	DHCP 	Internet with dynamic IP address.
== Network Interfaces ==
If there are multiple static IP addresses, or to be exact, if there are multiple default gateways, you must specify a metric value. The gateway with the lowest metric will win.
/etc/network/interfaces:
 auto eth0
 iface eth0 inet static
 address 1.2.3.4
 netmask 255.255.255.0
 gateway 1.2.3.1
 metric 200
The DHCP client will automatically add a metric value to the default gateway. It will pick 200 + interface index. Therefore we don't need to worry about that for DHCP.
 auto eth3
 iface eth0 inet dhcp
For PPP interfaces there is a defaultroute-metric option that can set the metric value.
==Shorewall Zones==
In /etc/shorewall/zones, create the zones:
 INET ipv4
 DMZ ipv4
 LAN ipv4
== Shorewall Interfaces ==
We need to make shorewall aware of the second ISP interface.
In /etc/shorewall/params, add ISP1 under ISP2.
 ISP1_IF=eth0
 ISP2_IF=eth3
 DMZ_IF=eth1
 LAN_IF=eth2
Add it to /etc/shorewall/interfaces:
 INET	$ISP1_IF	detect	dhcp
 INET	$ISP2_IF	detect	dhcp
 DMZ	$DMZ_IF	 detect	dhcp
 LAN	$LAN_IF	 detect	dhcp
The dhcp option is not really needed unless dhcp is actually used. We recommend leaving the dhcp option on in case the ISP changes to DHCP.
== Shorewall Masquerade ==
We must masquerade the internet addresses so that every ISP gets a correct source address. This is done in /etc/shorewall/masq.
 $ISP1_IF	0.0.0.0/0
 $ISP2_IF	0.0.0.0/0
== Shorewall Policy ==
We disallow routing between the 2 ISP's by adding a drop policy in /etc/shorewall/policy.
 ...
 INET INET DROP
 ...
== Shorewall Providers ==
The /etc/shorewall/providers defines the 2 ISP's and allows firewall packet marking (the firewall can mark certain packages which can later be used for the routing decision). Shorewall will also let the firewall mark the incomming connections in order to keep track of what gateway the response should go via. This is needed in case you do DNAT of incomming traffic on the secondary ISP. Without the connection tracking the outgoing traffic would always go via the primary ISP.
We do not want shorewall to mess with the routing tables (which we needed it to do with older versions of pingu). We can tell shorewall not to flush the routing tables that pingu will manage by setting DUPLICATE column to '-', GATEWAY column to 'none' and leaving the COPY column empty. 
In this example eth0 is attatched to ISP1, eth3 is attatched to ISP2.
 #NAME NUMBER MARK DUPLICATE INTERFACE GATEWAY OPTIONS COPY
 ISP1	1 1 - eth0 none track,optional,loose
 ISP2	2 2 - eth3 none track,optional,loose
After shorewall has restarted it should be possible to see the route rules for the firewall marks using the command ip rule:
 ...
 10001:	from all fwmark 0x1 lookup ISP1 
 10002:	from all fwmark 0x2 lookup ISP2 
 ...
== Shorewall Route Rules ==
We also use /etc/shorewall/route_rules to set up some static route rules. This is to force local traffic to ignore the alternate route tables for the ISP's.
 #SOURCE			DEST			PROVIDER	PRIORITY
 # prevent local traffic go via providers' route tables
 -			192.168.0.1/24		main		1000
 -			192.168.1.1/24		main		1000
== Make Shorewall not touch default gateway ==
Shorewall will store the default gateway(s) on startup and try restore on stopping. Unfortunally it does it badly, restulting in a default gateway getting lost on stop. Additionally, the default gateway might have changed during runtime, due to dhcp lease renewal. We need configure shorewall to not do anything with the default gateways at all and let pingu handle this.
Add the following to /etc/shorewall/shorewall.conf:
 RESTORE_DEFAULT_ROUTE=No
== Policy Routing with Pingu ==
We need to make sure that where ISP1's IP address is used as source address, the ISP1's default gateway must be used. Otherwise we might send packets with ISP1 IP source address via ISP2. The response would come through the ISP2's interfaces and firewall would block.
Since IP address and default gateway might change (DHCP lease renewal) we need a daemon that monitors the source address of the gateways via the interfaces. This is what pingu does.
 
 apk add pingu
We configure pingu to monitor our eth0 and eth3 interfaces in /etc/pingu/pingu.conf:
 interface eth0 { 
 # route-table must correspond with NUMBER column in /etc/shorewall/providers
 route-table 1
 # the rule-priority must be a higher number than the priority in /etc/shorewall/route_rules
 rule-priority 20000
 }
 interface eth3 {
 # route-table must correspond with NUMBER column in /etc/shorewall/providers
 route-table 2
 rule-priority 20000
 }
When pingu is started you should be able to see the routing rules with
 
 ip rule
 ...
 20000:	from 1.2.3.4 lookup ISP1 
 20000:	from 192.168.254.4 lookup ISP2 
 ...
and see the ISP route tables with
 ip route show table 1
 default via 1.2.3.1 dev eth0 metric 200
 1.2.3.0/24 dev eth0 proto kernel scope link metric 200 
== Internet Failover with Pingu ==
To do failover we need a way to detect if an ISP is still or not. We configure some hosts that pingu will ping for us at regular intervals. When a host stops to respond we consider that host offline. We can bind a ping host to an interface in pingu and when enough hosts stops to respond, pingu will remove that gateway from main route table and a gateway will lower metric will take over. We normally want ping more than one host since a single host might down without it meaning that the rest of internet is down.
We also want pick hosts that is unlikely to change. DNS servers are good in this aspect. They seldomly change since it would require all clients to do reconfiguration. In this example we will use google's DNS and opendns hosts.
Add the hosts to /etc/pingu/pingu.conf and bind to primary ISP. We also set the ping timeout to 2.5 seconds.
 ...
 # Ping responses that takes more than 2.5 seconds are considered lost.
 timeout 2.5
 # ping google dns via ISP1
 host 8.8.8.8 {
 interval 60
 bind-interface eth0
 }
 # ping opendns via ISP1
 host 208.67.222.222 {
 interval 60
 bind-interface eth0
 }
Now, if both hosts stops to respond to pings, ISP-1 will be considered down and all gateways via eth0 will be removed from main route table. Note that the gateway will not be removed from the route table '1'. This is so we can continue try ping via eth0 so we can detect that the ISP is back online. When ISP starts working again, the gateways will be added back to main route table again.
== Load Balancing ==
It is possible to give the impression of load balancing by using multipath routes. It will not let a http file download use both ISPs for double download speed but it will let some users surf via ISP1 and others via ISP2, resulting that bandwith usage is spread over both ISPs.
Note that balancing will not be perfect, as it is route based, and routes are cached. This means that routes to often-used sites will always be over the same provider. It also means that the source IP might change so websites that requires login (webmail, banks etc) will break if they check if the clients ip is consistent. Banking sites often do this. It is recommended to avoid this feature if possible.
To enable load balance, simply add the 'load-balance' keyword to all the interfaces that should be balanced in /etc/pingu/pingu.conf:
 interface eth0 {
 ...
 load-balance
 }
 interface eth3 {
 ...
 load-balance
 }
 ...
Category:Networking