Chapter 35. Getting started with Multipath TCP Transmission Control Protocol (TCP) ensures reliable delivery of the data through the internet and automatically adjusts its bandwidth in response to network load. Multipath TCP (MPTCP) is an extension to the original TCP protocol (single-path). MPTCP enables a transport connection to operate across multiple paths simultaneously, and brings network connection redundancy to user endpoint devices. 35.1. Understanding MPTCP The Multipath TCP (MPTCP) protocol allows for simultaneous usage of multiple paths between connection endpoints. The protocol design improves connection stability and also brings other benefits compared to the single-path TCP. Note In MPTCP terminology, links are considered as paths. The following are some of the advantages of using MPTCP: It allows a connection to simultaneously use multiple network interfaces. In case a connection is bound to a link speed, the usage of multiple links can increase the connection throughput. Note, that in case of the connection is bound to a CPU, the usage of multiple links causes the connection slowdown. It increases the resilience to link failures. For more details about MPTCP, review the Additional resources . Additional resources Understanding Multipath TCP: High availability for endpoints and the networking highway of the future RFC8684: TCP Extensions for Multipath Operation with Multiple Addresses Multipath TCP on Red Hat Enterprise Linux 8.3: From 0 to 1 subflows 35.2. Preparing RHEL to enable MPTCP support By default the MPTCP support is disabled in RHEL. Enable MPTCP so that applications that support this feature can use it. Additionally, you have to configure user space applications to force use MPTCP sockets if those applications have TCP sockets by default. You can use the sysctl utility to enable MPTCP support and prepare RHEL for enabling MPTCP for applications system-wide using a SystemTap script. Prerequisites The following packages are installed: systemtap iperf3 Procedure Enable MPTCP sockets in the kernel: Verify that MPTCP is enabled in the kernel: Create a mptcp-app.stap file with the following content: #!/usr/bin/env stap %{ #include <linux/in.h> #include <linux/ip.h> %} /* RSI contains 'type' and RDX contains 'protocol'. */ function mptcpify () %{ if (CONTEXT->kregs->si == SOCK_STREAM && (CONTEXT->kregs->dx == IPPROTO_TCP || CONTEXT->kregs->dx == 0)) { CONTEXT->kregs->dx = IPPROTO_MPTCP; STAP_RETVALUE = 1; } else { STAP_RETVALUE = 0; } %} probe kernel.function("__sys_socket") { if (mptcpify() == 1) { printf("command %16s mptcpified\n", execname()); } } Force user space applications to create MPTCP sockets instead of TCP ones: Note: This operation affects all TCP sockets which are started after the command. The applications will continue using TCP sockets after you interrupt the command above with Ctrl + C . Alternatively, to allow MPTCP usage to only specific application, you can modify the mptcp-app.stap file with the following content: #!/usr/bin/env stap %{ #include <linux/in.h> #include <linux/ip.h> %} /* according to [1], RSI contains 'type' and RDX * contains 'protocol'. * [1] https://github.com/torvalds/linux/blob/master/arch/x86/entry/entry_64.S#L79 */ function mptcpify () %{ if (CONTEXT->kregs->si == SOCK_STREAM && (CONTEXT->kregs->dx == IPPROTO_TCP || CONTEXT->kregs->dx == 0)) { CONTEXT->kregs->dx = IPPROTO_MPTCP; STAP_RETVALUE = 1; } else { STAP_RETVALUE = 0; } %} probe kernel.function("__sys_socket") { cur_proc = execname() if ((cur_proc == @1) && (mptcpify() == 1)) { printf("command %16s mptcpified\n", cur_proc); } } In case of alternative choice, assuming, you want to force the iperf3 tool to use MPTCP instead of TCP. To do so, enter the following command: After the mptcp-app.stap script installs the kernel probe, the following warnings appear in the kernel dmesg output Start the iperf3 server: Connect the client to the server: After the connection is established, verify the ss output to see the subflow-specific status: Verify MPTCP counters: Additional resources How can I download or install debuginfo packages for RHEL systems? (Red Hat Knowledgebase) tcp(7) and mptcpize(8) man pages on your system 35.3. Using iproute2 to temporarily configure and enable multiple paths for MPTCP applications Each MPTCP connection uses a single subflow similar to plain TCP. To get the MPTCP benefits, specify a higher limit for maximum number of subflows for each MPTCP connection. Then configure additional endpoints to create those subflows. Important The configuration in this procedure will not persist after rebooting your machine. Note that MPTCP does not yet support mixed IPv6 and IPv4 endpoints for the same socket. Use endpoints belonging to the same address family. Prerequisites The iperf3 package is installed Server network interface settings: enp4s0: 192.0.2.1/24 enp1s0: 198.51.100.1/24 Client network interface settings: enp4s0f0: 192.0.2.2/24 enp4s0f1: 198.51.100.2/24 Procedure Configure the client to accept up to 1 additional remote address, as provided by the server: Add IP address 198.51.100.1 as a new MPTCP endpoint on the server: The signal option ensures that the ADD_ADDR packet is sent after the three-way-handshake. Start the iperf3 server: Connect the client to the server: Verification Verify the connection is established: Verify the connection and IP address limit: Verify the newly added endpoint: Verify MPTCP counters by using the nstat MPTcp* command on a server: Additional resources mptcpize(8) and ip-mptcp(8) man pages on your system 35.4. Permanently configuring multiple paths for MPTCP applications You can configure MultiPath TCP (MPTCP) using the nmcli command to permanently establish multiple subflows between a source and destination system. The subflows can use different resources, different routes to the destination, and even different networks. Such as Ethernet, cellular, wifi, and so on. As a result, you achieve combined connections, which increase network resilience and throughput. The server uses the following network interfaces in our example: enp4s0: 192.0.2.1/24 enp1s0: 198.51.100.1/24 enp7s0: 192.0.2.3/24 The client uses the following network interfaces in our example: enp4s0f0: 192.0.2.2/24 enp4s0f1: 198.51.100.2/24 enp6s0: 192.0.2.5/24 Prerequisites You configured the default gateway on the relevant interfaces. Procedure Enable MPTCP sockets in the kernel: Optional: The RHEL kernel default for subflow limit is 2. If you require more: Create the /etc/systemd/system/set_mptcp_limit.service file with the following content: The oneshot unit executes the ip mptcp limits set subflows 3 command after your network ( network.target ) is operational during every boot process. The ip mptcp limits set subflows 3 command sets the maximum number of additional subflows for each connection, so 4 in total. It is possible to add maximally 3 additional subflows. Enable the set_mptcp_limit service: Enable MPTCP on all connection profiles that you want to use for connection aggregation: The connection.mptcp-flags parameter configures MPTCP endpoints and the IP address flags. If MPTCP is enabled in a NetworkManager connection profile, the setting will configure the IP addresses of the relevant network interface as MPTCP endpoints. By default, NetworkManager does not add MPTCP flags to IP addresses if there is no default gateway. If you want to bypass that check, you need to use the also-without-default-route flag. Verification Verify that you enabled the MPTCP kernel parameter: Verify that you set the subflow limit correctly, in case the default was not enough: Verify that you configured the per-address MPTCP setting correctly: Additional resources nm-settings-nmcli(5) ip-mptcp(8) Section 35.1, "Understanding MPTCP" Understanding Multipath TCP: High availability for endpoints and the networking highway of the future RFC8684: TCP Extensions for Multipath Operation with Multiple Addresses Using Multipath TCP to better survive outages and increase bandwidth 35.5. Monitoring MPTCP sub-flows The life cycle of a multipath TCP (MPTCP) socket can be complex: The main MPTCP socket is created, the MPTCP path is validated, one or more sub-flows are created and eventually removed. Finally, the MPTCP socket is terminated. The MPTCP protocol allows monitoring MPTCP-specific events related to socket and sub-flow creation and deletion, using the ip utility provided by the iproute package. This utility uses the netlink interface to monitor MPTCP events. This procedure demonstrates how to monitor MPTCP events. For that, it simulates a MPTCP server application, and a client connects to this service. The involved clients in this example use the following interfaces and IP addresses: Server: 192.0.2.1 Client (Ethernet connection): 192.0.2.2 Client (WiFi connection): 192.0.2.3 To simplify this example, all interfaces are within the same subnet. This is not a requirement. However, it is important that routing has been configured correctly, and the client can reach the server via both interfaces. Prerequisites A RHEL client with two network interfaces, such as a laptop with Ethernet and WiFi The client can connect to the server via both interfaces A RHEL server Both the client and the server run RHEL 8.6 or later Procedure Set the per connection additional subflow limits to 1 on both client and server: On the server, to simulate a MPTCP server application, start netcat ( nc ) in listen mode with enforced MPTCP sockets instead of TCP sockets: The -k option causes that nc does not close the listener after the first accepted connection. This is required to demonstrate the monitoring of sub-flows. On the client: Identify the interface with the lowest metric: The enp1s0 interface has a lower metric than wlp1s0 . Therefore, RHEL uses enp1s0 by default. On the first terminal, start the monitoring: On the second terminal, start a MPTCP connection to the server: RHEL uses the enp1s0 interface and its associated IP address as a source for this connection. On the monitoring terminal, the ip mptcp monitor command now logs: The token identifies the MPTCP socket as an unique ID, and later it enables you to correlate MPTCP events on the same socket. On the terminal with the running nc connection to the server, press Enter . This first data packet fully establishes the connection. Note that, as long as no data has been sent, the connection is not established. On the monitoring terminal, ip mptcp monitor now logs: Optional: Display the connections to port 12345 on the server: At this point, only one connection to the server has been established. On a third terminal, create another endpoint: This command sets the name and IP address of the WiFi interface of the client in this command. On the monitoring terminal, ip mptcp monitor now logs: The locid field displays the local address ID of the new sub-flow and identifies this sub-flow even if the connection uses network address translation (NAT). The saddr4 field matches the endpoint's IP address from the ip mptcp endpoint add command. Optional: Display the connections to port 12345 on the server: The command now displays two connections: The connection with source address 192.0.2.2 corresponds to the first MPTCP sub-flow that you established previously. The connection from the sub-flow over the wlp1s0 interface with source address 192.0.2.3 . On the third terminal, delete the endpoint: Use the ID from the locid field from the ip mptcp monitor output, or retrieve the endpoint ID using the ip mptcp endpoint show command. On the monitoring terminal, ip mptcp monitor now logs: On the first terminal with the nc client, press Ctrl + C to terminate the session. On the monitoring terminal, ip mptcp monitor now logs: Additional resources ip-mptcp(1) man page on your system How NetworkManager manages multiple default gateways 35.6. Disabling Multipath TCP in the kernel You can explicitly disable the MPTCP option in the kernel. Procedure Disable the mptcp.enabled option. Verification Verify whether the mptcp.enabled is disabled in the kernel.

Commands:
echo "net.mptcp.enabled=1" > /etc/sysctl.d/90-enable-MPTCP.conf sysctl -p /etc/sysctl.d/90-enable-MPTCP.conf
sysctl -a | grep mptcp.enabled net.mptcp.enabled = 1
#!/usr/bin/env stap %{ #include <linux/in.h> #include <linux/ip.h> %} /* RSI contains 'type' and RDX contains 'protocol'. */ function mptcpify () %{ if (CONTEXT->kregs->si == SOCK_STREAM && (CONTEXT->kregs->dx == IPPROTO_TCP || CONTEXT->kregs->dx == 0)) { CONTEXT->kregs->dx = IPPROTO_MPTCP; STAP_RETVALUE = 1; } else { STAP_RETVALUE = 0; } %} probe kernel.function("__sys_socket") { if (mptcpify() == 1) { printf("command %16s mptcpified\n", execname()); } }
stap -vg mptcp-app.stap
#!/usr/bin/env stap %{ #include <linux/in.h> #include <linux/ip.h> %} /* according to [1], RSI contains 'type' and RDX * contains 'protocol'. * [1] https://github.com/torvalds/linux/blob/master/arch/x86/entry/entry_64.S#L79 */ function mptcpify () %{ if (CONTEXT->kregs->si == SOCK_STREAM && (CONTEXT->kregs->dx == IPPROTO_TCP || CONTEXT->kregs->dx == 0)) { CONTEXT->kregs->dx = IPPROTO_MPTCP; STAP_RETVALUE = 1; } else { STAP_RETVALUE = 0; } %} probe kernel.function("__sys_socket") { cur_proc = execname() if ((cur_proc == @1) && (mptcpify() == 1)) { printf("command %16s mptcpified\n", cur_proc); } }
stap -vg mptcp-app.stap iperf3
dmesg [ 1752.694072] Kprobes globally unoptimized [ 1752.730147] stap_1ade3b3356f3e68765322e26dec00c3d_1476: module_layout: kernel tainted. [ 1752.732162] Disabling lock debugging due to kernel taint [ 1752.733468] stap_1ade3b3356f3e68765322e26dec00c3d_1476: loading out-of-tree module taints kernel. [ 1752.737219] stap_1ade3b3356f3e68765322e26dec00c3d_1476: module verification failed: signature and/or required key missing - tainting kernel [ 1752.737219] stap_1ade3b3356f3e68765322e26dec00c3d_1476 (mptcp-app.stap): systemtap: 4.5/0.185, base: ffffffffc0550000, memory: 224data/32text/57ctx/65638net/367alloc kb, probes: 1
iperf3 -s Server listening on 5201
iperf3 -c 127.0.0.1 -t 3
ss -nti '( dport :5201 )' State Recv-Q Send-Q Local Address:Port Peer Address:Port Process ESTAB 0 0 127.0.0.1:41842 127.0.0.1:5201 cubic wscale:7,7 rto:205 rtt:4.455/8.878 ato:40 mss:21888 pmtu:65535 rcvmss:536 advmss:65483 cwnd:10 bytes_sent:141 bytes_acked:142 bytes_received:4 segs_out:8 segs_in:7 data_segs_out:3 data_segs_in:3 send 393050505bps lastsnd:2813 lastrcv:2772 lastack:2772 pacing_rate 785946640bps delivery_rate 10944000000bps delivered:4 busy:41ms rcv_space:43690 rcv_ssthresh:43690 minrtt:0.008 tcp-ulp-mptcp flags:Mmec token:0000(id:0)/2ff053ec(id:0) seq:3e2cbea12d7673d4 sfseq:3 ssnoff:ad3d00f4 maplen:2
nstat MPTcp * #kernel MPTcpExtMPCapableSYNRX 2 0.0 MPTcpExtMPCapableSYNTX 2 0.0 MPTcpExtMPCapableSYNACKRX 2 0.0 MPTcpExtMPCapableACKRX 2 0.0
ip mptcp limits set add_addr_accepted 1
ip mptcp endpoint add 198.51.100.1 dev enp1s0 signal
iperf3 -s Server listening on 5201
iperf3 -c 192.0.2.1 -t 3
ss -nti '( sport :5201 )'
ip mptcp limit show
ip mptcp endpoint show
nstat MPTcp * #kernel MPTcpExtMPCapableSYNRX 2 0.0 MPTcpExtMPCapableACKRX 2 0.0 MPTcpExtMPJoinSynRx 2 0.0 MPTcpExtMPJoinAckRx 2 0.0 MPTcpExtEchoAdd 2 0.0
echo "net.mptcp.enabled=1" > /etc/sysctl.d/90-enable-MPTCP.conf sysctl -p /etc/sysctl.d/90-enable-MPTCP.conf
[Unit] Description=Set MPTCP subflow limit to 3 After=network.target [Service] ExecStart=ip mptcp limits set subflows 3 Type=oneshot [Install] WantedBy=multi-user.target
systemctl enable --now set_mptcp_limit
nmcli connection modify <profile_name> connection.mptcp-flags signal,subflow,also-without-default-route
sysctl net.mptcp.enabled net.mptcp.enabled = 1
ip mptcp limit show add_addr_accepted 2 subflows 3
ip mptcp endpoint show 192.0.2.1 id 1 subflow dev enp4s0 198.51.100.1 id 2 subflow dev enp1s0 192.0.2.3 id 3 subflow dev enp7s0 192.0.2.4 id 4 subflow dev enp3s0
ip mptcp limits set add_addr_accepted 0 subflows 1
nc -l -k -p 12345
ip -4 route 192.0.2.0/24 dev enp1s0 proto kernel scope link src 192.0.2.2 metric 100 192.0.2.0/24 dev wlp1s0 proto kernel scope link src 192.0.2.3 metric 600
ip mptcp monitor
nc 192.0.2.1 12345
[ CREATED] token=63c070d2 remid=0 locid=0 saddr4=192.0.2.2 daddr4=192.0.2.1 sport=36444 dport=12345
[ ESTABLISHED] token=63c070d2 remid=0 locid=0 saddr4=192.0.2.2 daddr4=192.0.2.1 sport=36444 dport=12345
ss -taunp | grep ":12345" tcp ESTAB 0 0 192.0.2.2:36444 192.0.2.1:12345
ip mptcp endpoint add dev wlp1s0 192.0.2.3 subflow
[SF_ESTABLISHED] token=63c070d2 remid=0 locid=2 saddr4=192.0.2.3 daddr4=192.0.2.1 sport=53345 dport=12345 backup=0 ifindex=3
ss -taunp | grep ":12345" tcp ESTAB 0 0 192.0.2.2:36444 192.0.2.1:12345 tcp ESTAB 0 0 192.0.2.3%wlp1s0:53345 192.0.2.1:12345
ip mptcp endpoint delete id 2
[ SF_CLOSED] token=63c070d2 remid=0 locid=2 saddr4=192.0.2.3 daddr4=192.0.2.1 sport=53345 dport=12345 backup=0 ifindex=3
[ CLOSED] token=63c070d2
echo "net.mptcp.enabled=0" > /etc/sysctl.d/90-enable-MPTCP.conf sysctl -p /etc/sysctl.d/90-enable-MPTCP.conf
sysctl -a | grep mptcp.enabled net.mptcp.enabled = 0