Repository: firezone/firezone
Pull request #8731: feat(connlib): request larger buffers for UDP sockets
Author: login: thomaseizinger
Created: 2025-04-09T02:51:30Z | Merged: 2025-04-22T07:11:03Z

Sufficiently large receive buffers are important to sustain high-throughput as latency increases. If the receive buffer in the kernel is too small, packets need to be dropped on arrival.

Firefox uses 1MB in its QUIC stack [0]. `quic-go` recommends to set send and receive buffers to 7.5 MB [1]. Power users of Firezone are likely receiving a lot more traffic than the average Firefox user (especially with Internet Resource activated) so setting it to 10 MB seems reasonable. Sending packets is likely not as critical because we have back-pressure through our system such that we will stop reading IP packets when we cannot write to our UDP socket. The UDP socket is sitting in a separate thread and those threads are connected with dedicated queues which act as another buffer. However, as the data below shows, some systems have really small send buffers which are currently likely a speed bottleneck because we need to suspend writing so frequently.

Assuming a 50ms latency, the bandwidth-delay product tells us that we can (in theory) saturate a 1.6 Gbps link with a 10MB receive buffer (assuming the OS also has large enough buffer sizes in its TCP or QUIC stack):

```
80 Mb / 0.05s = 1600Mbps
```

Experiments and research [2] show the following:

|OS|Receive buffer (default)|Receive buffer (this PR)|Send buffer (default)|Send buffer (this PR)|
|---|---|---|---|---|
|Windows|65KB|10MB|65KB|1MB|
|MacOS|786KB|8MB|9KB|1MB|
|Linux|212KB|212KB|212KB|212KB|

With the exception of Linux, the OSes appear to be quite generous with how big they allow receive buffers to be. On Linux, these limit can be changed by setting the `core.net.rmem_max` and `core.net.wmem_max` parameters using `sysctl`.

Most of our users are on Windows and MacOS, meaning they immediately benefit from this without having to change any system settings. Larger client-side UDP receive buffers are critical for any "download" scenario which is likely the majority of usecases that Firezone is used for.

On Windows, increasing this receive buffer almost doubles the throughput in an iperf3 download test.

[0]: https://github.com/mozilla/neqo/pull/2470
[1]: https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes
[2]: https://unix.stackexchange.com/a/424381


Diff:
diff --git a/rust/connlib/tunnel/src/sockets.rs b/rust/connlib/tunnel/src/sockets.rs
index 77bfb36c59f..5f344e28b69 100644
--- a/rust/connlib/tunnel/src/sockets.rs
+++ b/rust/connlib/tunnel/src/sockets.rs
@@ -173,7 +173,7 @@ impl ThreadedUdpSocket {
                     .build()
                     .expect("Failed to spawn tokio runtime on UDP thread")
                     .block_on(async move {
-                        let socket = match sf(&addr) {
+                        let mut socket = match sf(&addr) {
                             Ok(s) => {
                                 let _ = error_tx.send(Ok(()));
 
@@ -185,6 +185,13 @@ impl ThreadedUdpSocket {
                             }
                         };
 
+                        match socket.set_buffer_sizes(socket_factory::SEND_BUFFER_SIZE, socket_factory::RECV_BUFFER_SIZE) {
+                            Ok(()) => {},
+                            Err(e) => {
+                                let _ = error_tx.send(Err(e));
+                                return;
+                            },
+                        }
 
                         let send = pin!(async {
                             while let Ok(datagram) = outbound_rx.recv_async().await {
diff --git a/rust/socket-factory/src/lib.rs b/rust/socket-factory/src/lib.rs
index 4404cce9b17..d719b0e9a7c 100644
--- a/rust/socket-factory/src/lib.rs
+++ b/rust/socket-factory/src/lib.rs
@@ -23,6 +23,10 @@ use tokio::io::Interest;
 
 pub trait SocketFactory<S>: Fn(&SocketAddr) -> io::Result<S> + Send + Sync + 'static {}
 
+pub const SEND_BUFFER_SIZE: usize = ONE_MB;
+pub const RECV_BUFFER_SIZE: usize = 10 * ONE_MB;
+const ONE_MB: usize = 1024 * 1024;
+
 impl<F, S> SocketFactory<S> for F where F: Fn(&SocketAddr) -> io::Result<S> + Send + Sync + 'static {}
 
 pub fn tcp(addr: &SocketAddr) -> io::Result<TcpSocket> {
@@ -183,6 +187,28 @@ impl UdpSocket {
         })
     }
 
+    pub fn set_buffer_sizes(
+        &mut self,
+        requested_send_buffer_size: usize,
+        requested_recv_buffer_size: usize,
+    ) -> io::Result<()> {
+        let socket = socket2::SockRef::from(&self.inner);
+
+        socket.set_send_buffer_size(requested_send_buffer_size)?;
+        socket.set_recv_buffer_size(requested_recv_buffer_size)?;
+
+        let send_buffer_size = socket.send_buffer_size()?;
+        let recv_buffer_size = socket.recv_buffer_size()?;
+
+        tracing::info!(%requested_send_buffer_size, %send_buffer_size, %requested_recv_buffer_size, %recv_buffer_size, port = %self.port, "Set UDP socket buffer sizes");
+
+        Ok(())
+    }
+
+    pub fn port(&self) -> u16 {
+        self.port
+    }
+
     /// Configures a new source IP resolver for this UDP socket.
     ///
     /// In case [`DatagramOut::src`] is [`None`], this function will be used to set a source IP given the destination IP of the datagram.
diff --git a/website/src/app/kb/deploy/gateways/readme.mdx b/website/src/app/kb/deploy/gateways/readme.mdx
index e0db8a44fd9..6d6870d5e0a 100644
--- a/website/src/app/kb/deploy/gateways/readme.mdx
+++ b/website/src/app/kb/deploy/gateways/readme.mdx
@@ -125,6 +125,17 @@ Firezone's
 [automatic load balancing](/kb/architecture/critical-sequences#high-availability)
 to distribute Client connections across them.
 
+### Performance tuning
+
+The default receive buffer size on Linux is quite small which can limit the
+maximum throughput that users perceive in "upload scenarios" (i.e. where the
+Gateway needs to receive large volumes of traffic).
+
+On startup, the Gateway attempts to increase the size of the UDP receive buffers
+to 10 MB. However, the actual size of the receive buffer is limited by the
+`net.core.rmem_max` kernel parameter. For the increased buffer size to take
+effect, you may need to increase the `net.core.rmem_max` parameter on the
+Gateway's host system.
 ## Deploy a single Gateway
 
 Deploying a single Gateway can be accomplished in the admin portal.
diff --git a/website/src/components/Changelog/Apple.tsx b/website/src/components/Changelog/Apple.tsx
index 9c01362277c..4d10dae8106 100644
--- a/website/src/components/Changelog/Apple.tsx
+++ b/website/src/components/Changelog/Apple.tsx
@@ -23,7 +23,14 @@ export default function Apple() {
   return (
     <Entries downloadLinks={downloadLinks} title="macOS / iOS">
       {/* When you cut a release, remove any solved issues from the "known issues" lists over in `client-apps`. This must not be done when the issue's PR merges. */}
-      <Unreleased></Unreleased>
+      <Unreleased>
+        <ChangeItem pull="8731">
+          Improves throughput performance by requesting socket receive buffers
+          of 10MB. The actual size of the buffers is capped by the operating
+          system. You may need to adjust <code>kern.ipc.maxsockbuf</code> for this to take
+          full effect.
+        </ChangeItem>
+      </Unreleased>
       <Entry version="1.4.12" date={new Date("2025-04-21")}>
         <ChangeItem pull="8798">
           Improves performance of relayed connections on IPv4-only systems.
diff --git a/website/src/components/Changelog/GUI.tsx b/website/src/components/Changelog/GUI.tsx
index fc1b4b149eb..8380eba0541 100644
--- a/website/src/components/Changelog/GUI.tsx
+++ b/website/src/components/Changelog/GUI.tsx
@@ -8,7 +8,22 @@ export default function GUI({ os }: { os: OS }) {
   return (
     <Entries downloadLinks={downloadLinks(os)} title={title(os)}>
       {/* When you cut a release, remove any solved issues from the "known issues" lists over in `client-apps`. This must not be done when the issue's PR merges. */}
-      <Unreleased></Unreleased>
+      <Unreleased>
+        {os === OS.Linux && (
+          <ChangeItem pull="8731">
+            Improves throughput performance by requesting socket receive buffers
+            of 10MB. The actual size of the buffers is capped by the operating
+            system. You may need to adjust <code>net.core.rmem_max</code> for this to take
+            full effect.
+          </ChangeItem>
+          )}
+        {os === OS.Windows && (
+          <ChangeItem pull="8731">
+            Improves throughput performance by requesting socket receive buffers
+            of 10MB.
+          </ChangeItem>
+          )}
+      </Unreleased>
       <Entry version="1.4.11" date={new Date("2025-04-21")}>
         <ChangeItem pull="8798">
           Improves performance of relayed connections on IPv4-only systems.
diff --git a/website/src/components/Changelog/Gateway.tsx b/website/src/components/Changelog/Gateway.tsx
index 5bd1a5bb33a..485d5be2635 100644
--- a/website/src/components/Changelog/Gateway.tsx
+++ b/website/src/components/Changelog/Gateway.tsx
@@ -26,6 +26,12 @@ export default function Gateway() {
         <ChangeItem pull="8798">
           Improves performance of relayed connections on IPv4-only systems.
         </ChangeItem>
+        <ChangeItem pull="8731">
+          Improves throughput performance by requesting socket receive buffers
+          of 10MB. The actual size of the buffers is capped by the operating
+          system. You may need to adjust <code>net.core.rmem_max</code> for this to take
+          full effect.
+        </ChangeItem>
       </Unreleased>
       <Entry version="1.4.6" date={new Date("2025-04-15")}>
         <ChangeItem pull="8383">
diff --git a/website/src/components/Changelog/Headless.tsx b/website/src/components/Changelog/Headless.tsx
index f82aff5a7b4..abac1ddf0ca 100644
--- a/website/src/components/Changelog/Headless.tsx
+++ b/website/src/components/Changelog/Headless.tsx
@@ -13,6 +13,20 @@ export default function Headless({ os }: { os: OS }) {
         <ChangeItem pull="8798">
           Improves performance of relayed connections on IPv4-only systems.
         </ChangeItem>
+        {os === OS.Linux && (
+          <ChangeItem pull="8731">
+            Improves throughput performance by requesting socket receive buffers
+            of 10MB. The actual size of the buffers is capped by the operating
+            system. You may need to adjust <code>net.core.rmem_max</code> for this to take
+            full effect.
+          </ChangeItem>
+          )}
+        {os === OS.Windows && (
+          <ChangeItem pull="8731">
+            Improves throughput performance by requesting socket receive buffers
+            of 10MB.
+          </ChangeItem>
+          )}
       </Unreleased>
       <Entry version="1.4.6" date={new Date("2025-04-15")}>
         {os == OS.Linux && (


Reviews:
user: login: jamilbk
body: Would it make a difference adjusting queue settings for the tun device as well?
state: DISMISSED
author_association: MEMBER
submitted_at: 2025-04-09T03:31:20Z
user: login: thomaseizinger
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-12T08:45:14Z
user: login: jamilbk
body: Is this for each UDP socket opened by the IPC service? I'll probably spend some time measuring memory usage increase.

Would that be expected to grow with the number of Gateways a user connects to?
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-21T13:10:28Z
user: login: jamilbk
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-21T13:15:45Z
user: login: thomaseizinger
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-04-21T13:17:31Z
user: login: jamilbk
state: APPROVED
author_association: MEMBER
submitted_at: 2025-04-22T03:04:02Z

Review comments:
diff_hunk: @@ -189,6 +189,10 @@ jobs:
           docker compose exec -d gateway tc qdisc add dev eth0 root netem delay 10ms
           docker compose exec -d relay-1 tc qdisc add dev eth0 root netem delay 10ms
           docker compose exec -d relay-2 tc qdisc add dev eth0 root netem delay 10ms
+      - name: Increase receive buffer sizes
path: .github/workflows/ci.yml
user: login: thomaseizinger
body: ```suggestion
      - name: Increase send & receive buffer sizes
```
created_at: 2025-04-12T08:45:13Z
author_association: MEMBER
diff_hunk: @@ -50,6 +54,9 @@ pub fn udp(std_addr: &SocketAddr) -> io::Result<UdpSocket> {
     socket.set_nonblocking(true)?;
     socket.bind(&addr)?;
 
+    socket.set_send_buffer_size(SEND_BUFFER_SIZE)?;
+    socket.set_recv_buffer_size(RECV_BUFFER_SIZE)?;
+
     let send_buf_size = socket.send_buffer_size()?;
     let recv_buf_size = socket.recv_buffer_size()?;
 
path: rust/socket-factory/src/lib.rs
user: login: jamilbk
body: Any value here you think in reporting this to Sentry so we have a rough idea of the actual size the OS has given us per platform?
created_at: 2025-04-21T13:15:45Z
author_association: MEMBER
diff_hunk: @@ -50,6 +54,9 @@ pub fn udp(std_addr: &SocketAddr) -> io::Result<UdpSocket> {
     socket.set_nonblocking(true)?;
     socket.bind(&addr)?;
 
+    socket.set_send_buffer_size(SEND_BUFFER_SIZE)?;
+    socket.set_recv_buffer_size(RECV_BUFFER_SIZE)?;
+
     let send_buf_size = socket.send_buffer_size()?;
     let recv_buf_size = socket.recv_buffer_size()?;
 
path: rust/socket-factory/src/lib.rs
user: login: thomaseizinger
body: We can change the log-level to DEBUG but it is quite spammy because we create a new UDP socket for each forwarded DNS query (to ensure we use randomized ports). It should be the same value for all sockets really so perhaps I can log them every time we rebind the p2p sockets.
created_at: 2025-04-21T13:17:31Z
author_association: MEMBER
diff_hunk: @@ -23,7 +23,14 @@ export default function Apple() {
   return (
     <Entries downloadLinks={downloadLinks} title="macOS / iOS">
       {/* When you cut a release, remove any solved issues from the "known issues" lists over in `client-apps`. This must not be done when the issue's PR merges. */}
-      <Unreleased></Unreleased>
+      <Unreleased>
+        <ChangeItem pull="8731">
+          Improves throughput performance by requesting socket receive buffers
+          of 10MB. The actual size of the buffers is capped by the operating
+          system. You may need to adjust `kern.ipc.maxsockbuf` for this to take
path: website/src/components/Changelog/Apple.tsx
user: login: jamilbk
body: ```suggestion
          system. You may need to adjust <code>kern.ipc.maxsockbuf</code> for this to take
```
created_at: 2025-04-22T02:59:41Z
author_association: MEMBER
diff_hunk: @@ -8,7 +8,22 @@ export default function GUI({ os }: { os: OS }) {
   return (
     <Entries downloadLinks={downloadLinks(os)} title={title(os)}>
       {/* When you cut a release, remove any solved issues from the "known issues" lists over in `client-apps`. This must not be done when the issue's PR merges. */}
-      <Unreleased></Unreleased>
+      <Unreleased>
+        {os === OS.Linux && (
+          <ChangeItem pull="8731">
+            Improves throughput performance by requesting socket receive buffers
+            of 10MB. The actual size of the buffers is capped by the operating
+            system. You may need to adjust `net.core.rmem_max` for this to take
path: website/src/components/Changelog/GUI.tsx
user: login: jamilbk
body: ```suggestion
            system. You may need to adjust <code>net.core.rmem_max</code> for this to take
```
created_at: 2025-04-22T03:00:13Z
author_association: MEMBER
diff_hunk: @@ -26,6 +26,12 @@ export default function Gateway() {
         <ChangeItem pull="8798">
           Improves performance of relayed connections on IPv4-only systems.
         </ChangeItem>
+        <ChangeItem pull="8731">
+          Improves throughput performance by requesting socket receive buffers
+          of 10MB. The actual size of the buffers is capped by the operating
+          system. You may need to adjust `net.core.rmem_max` for this to take
path: website/src/components/Changelog/Gateway.tsx
user: login: jamilbk
body: ```suggestion
          system. You may need to adjust <code>net.core.rmem_max</code> for this to take
```
created_at: 2025-04-22T03:00:33Z
author_association: MEMBER
diff_hunk: @@ -13,6 +13,20 @@ export default function Headless({ os }: { os: OS }) {
         <ChangeItem pull="8798">
           Improves performance of relayed connections on IPv4-only systems.
         </ChangeItem>
+        {os === OS.Linux && (
+          <ChangeItem pull="8731">
+            Improves throughput performance by requesting socket receive buffers
+            of 10MB. The actual size of the buffers is capped by the operating
+            system. You may need to adjust `net.core.rmem_max` for this to take
path: website/src/components/Changelog/Headless.tsx
user: login: jamilbk
body: ```suggestion
            system. You may need to adjust <code>net.core.rmem_max</code> for this to take
```
created_at: 2025-04-22T03:00:50Z
author_association: MEMBER

PR comments:
user: login: vercel[bot]
created_at: 2025-04-09T02:51:34Z
body: [vc]: #CdxsozRHakXO9hLtKZ2zufBvMmemB1E+hg01HDcjso8=:eyJpc01vbm9yZXBvIjp0cnVlLCJ0eXBlIjoiZ2l0aHViIiwicHJvamVjdHMiOlt7Im5hbWUiOiJmaXJlem9uZSIsImluc3BlY3RvclVybCI6Imh0dHBzOi8vdmVyY2VsLmNvbS9maXJlem9uZS9maXJlem9uZS9Cdzc3UlR0VHNvc0Z0R0RvQ3o5ZHBlNm9NbjViIiwicHJldmlld1VybCI6ImZpcmV6b25lLWdpdC1maXgtYnVtcC11ZHAtYnVmZmVyLXNpemUtZmlyZXpvbmUudmVyY2VsLmFwcCIsIm5leHRDb21taXRTdGF0dXMiOiJERVBMT1lFRCIsImxpdmVGZWVkYmFjayI6eyJyZXNvbHZlZCI6MCwidW5yZXNvbHZlZCI6MCwidG90YWwiOjAsImxpbmsiOiJmaXJlem9uZS1naXQtZml4LWJ1bXAtdWRwLWJ1ZmZlci1zaXplLWZpcmV6b25lLnZlcmNlbC5hcHAifSwicm9vdERpcmVjdG9yeSI6IndlYnNpdGUifV19
**The latest updates on your projects**. Learn more about [Vercel for Git ↗︎](https://vercel.link/github-learn-more)

| Name | Status | Preview | Comments | Updated (UTC) |
| :--- | :----- | :------ | :------- | :------ |
| **firezone** | ✅ Ready ([Inspect](https://vercel.com/firezone/firezone/Bw77RTtTsosFtGDoCz9dpe6oMn5b)) | [Visit Preview](https://firezone-git-fix-bump-udp-buffer-size-firezone.vercel.app) | 💬 [**Add feedback**](https://vercel.live/open-feedback/firezone-git-fix-bump-udp-buffer-size-firezone.vercel.app?via=pr-comment-feedback-link) | Apr 22, 2025 4:23am |


author_association: NONE
user: login: thomaseizinger
created_at: 2025-04-09T03:16:38Z
body: Can't measure any difference, at least no in CI :(
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-12T08:48:45Z
body: Once #8754 is merged, we should be able to see this finally taking effect. I could measure a difference here locally already.

I _don't_ think we should try and set this from within the binary. I know that current deployments of Gateway will not get this as a result but messing with system settings like this just doesn't seem like a good idea to do from within the binary.

I think a better long-term solution here is to offer OS packages for the Gateway, i.e. `.deb` and others. With those, we can ship updates to these scripts as part of new releases.
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-21T06:46:26Z
body: Interesing, with this patch, Windows directly hands me a UDP socket with a 10MB buffer size without changing any system settings. This almost doubles the throughput I am getting during a download:

Latest release:

```
PS C:\Users\thomas\Downloads\iperf-3.18-win64> .\iperf3.exe -c  10.6.96.4 -t 0
Connecting to host 10.6.96.4, port 5201
[  5] local 100.92.78.37 port 59737 connected to 10.6.96.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  46.5 MBytes   387 Mbits/sec
[  5]   1.01-2.02   sec  26.0 MBytes   216 Mbits/sec
[  5]   2.02-3.01   sec  14.2 MBytes   120 Mbits/sec
[  5]   3.01-4.01   sec  15.1 MBytes   126 Mbits/sec
[  5]   4.01-5.00   sec  18.1 MBytes   154 Mbits/sec
[  5]   5.00-6.00   sec  21.9 MBytes   184 Mbits/sec
[  5]   6.00-7.01   sec  27.6 MBytes   231 Mbits/sec
[  5]   7.01-8.02   sec  32.4 MBytes   269 Mbits/sec
[  5]   8.02-9.00   sec  36.9 MBytes   313 Mbits/sec
[  5]   9.00-10.01  sec  41.9 MBytes   350 Mbits/sec
[  5]  10.01-11.01  sec  46.2 MBytes   386 Mbits/sec
[  5]  11.01-12.00  sec  37.8 MBytes   320 Mbits/sec
[  5]  12.00-13.01  sec  42.9 MBytes   358 Mbits/sec
[  5]  13.01-14.01  sec  34.4 MBytes   286 Mbits/sec
[  5]  14.01-15.00  sec  33.6 MBytes   285 Mbits/sec
[  5]  15.00-16.00  sec  24.9 MBytes   208 Mbits/sec
[  5]  16.00-17.01  sec  27.1 MBytes   226 Mbits/sec
[  5]  17.01-18.00  sec  29.8 MBytes   252 Mbits/sec
[  5]  18.00-19.00  sec  33.6 MBytes   281 Mbits/sec
[  5]  19.00-20.01  sec  35.2 MBytes   295 Mbits/sec
[  5]  20.01-21.01  sec  41.9 MBytes   350 Mbits/sec
[  5]  21.01-22.00  sec  41.4 MBytes   350 Mbits/sec
[  5]  22.00-23.01  sec  35.6 MBytes   298 Mbits/sec
[  5]  23.01-24.01  sec  39.1 MBytes   326 Mbits/sec
[  5]  24.01-25.01  sec  39.9 MBytes   334 Mbits/sec
[  5]  25.01-26.01  sec  37.2 MBytes   312 Mbits/sec
[  5]  26.01-27.00  sec  33.0 MBytes   280 Mbits/sec
[  5]  27.00-28.01  sec  36.0 MBytes   301 Mbits/sec
[  5]  28.01-29.01  sec  17.6 MBytes   148 Mbits/sec
[  5]  29.01-30.01  sec  26.8 MBytes   224 Mbits/sec
[  5]  30.01-31.00  sec  30.0 MBytes   254 Mbits/sec
[  5]  31.00-32.01  sec  32.8 MBytes   274 Mbits/sec
[  5]  32.01-33.01  sec  36.1 MBytes   302 Mbits/sec
[  5]  33.01-33.91  sec  33.6 MBytes   314 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-33.91  sec  1.08 GBytes   274 Mbits/sec                  sender
[  5]   0.00-33.91  sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated
```

This PR:

```
PS C:\Users\thomas\Downloads\iperf-3.18-win64> .\iperf3.exe -c  10.6.96.4 -t 0
Connecting to host 10.6.96.4, port 5201
[  5] local 100.92.78.37 port 59569 connected to 10.6.96.4 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec  42.1 MBytes   351 Mbits/sec
[  5]   1.01-2.01   sec  48.5 MBytes   407 Mbits/sec
[  5]   2.01-3.01   sec  51.2 MBytes   431 Mbits/sec
[  5]   3.01-4.01   sec  39.9 MBytes   332 Mbits/sec
[  5]   4.01-5.01   sec  58.9 MBytes   496 Mbits/sec
[  5]   5.01-6.01   sec  73.0 MBytes   612 Mbits/sec
[  5]   6.01-7.01   sec  73.1 MBytes   614 Mbits/sec
[  5]   7.01-8.00   sec  62.1 MBytes   526 Mbits/sec
[  5]   8.00-9.01   sec  68.0 MBytes   564 Mbits/sec
[  5]   9.01-10.00  sec  59.0 MBytes   499 Mbits/sec
[  5]  10.00-11.01  sec  70.2 MBytes   586 Mbits/sec
[  5]  11.01-12.01  sec  69.6 MBytes   584 Mbits/sec
[  5]  12.01-13.01  sec  66.5 MBytes   560 Mbits/sec
[  5]  13.01-14.01  sec  63.9 MBytes   533 Mbits/sec
[  5]  14.01-15.01  sec  67.0 MBytes   561 Mbits/sec
[  5]  15.01-16.01  sec  67.8 MBytes   571 Mbits/sec
[  5]  16.01-17.00  sec  68.1 MBytes   576 Mbits/sec
[  5]  17.00-18.01  sec  70.2 MBytes   583 Mbits/sec
[  5]  18.01-19.01  sec  73.1 MBytes   616 Mbits/sec
[  5]  19.01-20.01  sec  68.8 MBytes   574 Mbits/sec
[  5]  20.01-21.00  sec  56.8 MBytes   481 Mbits/sec
[  5]  21.00-22.01  sec  63.4 MBytes   526 Mbits/sec
[  5]  22.01-23.00  sec  68.6 MBytes   581 Mbits/sec
[  5]  23.00-24.01  sec  67.0 MBytes   562 Mbits/sec
[  5]  24.01-25.01  sec  56.1 MBytes   470 Mbits/sec
[  5]  25.01-26.01  sec  70.9 MBytes   595 Mbits/sec
[  5]  26.01-27.01  sec  68.9 MBytes   577 Mbits/sec
[  5]  27.01-28.01  sec  67.5 MBytes   564 Mbits/sec
[  5]  28.01-29.02  sec  72.8 MBytes   608 Mbits/sec
[  5]  29.02-30.01  sec  64.0 MBytes   541 Mbits/sec
[  5]  30.01-31.01  sec  35.4 MBytes   297 Mbits/sec
[  5]  31.01-32.01  sec  64.5 MBytes   541 Mbits/sec
[  5]  32.01-33.01  sec  60.2 MBytes   506 Mbits/sec
[  5]  33.01-34.01  sec  61.4 MBytes   514 Mbits/sec
[  5]  34.01-35.01  sec  47.9 MBytes   401 Mbits/sec
[  5]  35.01-36.01  sec  61.4 MBytes   514 Mbits/sec
[  5]  36.01-37.00  sec  68.2 MBytes   579 Mbits/sec
[  5]  37.00-38.00  sec  59.2 MBytes   496 Mbits/sec
[  5]  38.00-39.00  sec  69.1 MBytes   580 Mbits/sec
[  5]  39.00-40.00  sec  66.1 MBytes   555 Mbits/sec
[  5]  40.00-41.00  sec  65.8 MBytes   552 Mbits/sec
[  5]  41.00-42.00  sec  66.2 MBytes   554 Mbits/sec
[  5]  42.00-43.01  sec  70.2 MBytes   588 Mbits/sec
[  5]  43.01-44.01  sec  72.6 MBytes   608 Mbits/sec
[  5]  44.01-45.01  sec  51.4 MBytes   431 Mbits/sec
[  5]  45.01-46.01  sec  62.1 MBytes   520 Mbits/sec
[  5]  46.01-46.77  sec  51.6 MBytes   573 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-46.77  sec  2.88 GBytes   529 Mbits/sec                  sender
[  5]   0.00-46.77  sec  0.00 Bytes  0.00 bits/sec                  receiver
iperf3: interrupt - the client has terminated
```
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-21T08:31:25Z
body: @jamilbk My suggestion is to ship this before we mess around with the operating system parameters. It doesn't feel quite right to adjust those from within our application.

I added some docs for the Gateway that setting `net.core.rmem_max` may help to increase performance.

I've left it out in the Clients so far as my guess is that most users won't bother with increasing those settings on their machines. For simplicity's sake, the `socket_factory` module is used by both Clients and Gateways and it didn't seem worth it to split that configuration up. In case a user has increased their OS settings or the defaults change, it will automatically apply for them.

At least on Windows, this immediately has a positive effect because the default buffer size there is _really_ small (only 65k) and it appears the maximum allowed size is quite large.
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-21T09:47:37Z
body: @jamilbk or I've built an Apple client with this here: https://github.com/firezone/firezone/actions/runs/14573525934/job/40874959638

Could you test if you get better download numbers with this on your connection? I can easily saturate my Internet prior to this PR already so it is difficult for me to see if there is an improvement on MacOS.
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-21T10:23:11Z
body: > @jamilbk or I've built an Apple client with this here: [firezone/firezone/actions/runs/14571295945/job/40868936348](https://github.com/firezone/firezone/actions/runs/14571295945/job/40868936348)
> 
> Could you test if you get better download numbers with this on your connection? I can easily saturate my Internet prior to this PR already so it is difficult for me to see if there is an improvement on MacOS.

It appears that #8795 broke the upload of these files: https://github.com/firezone/firezone/actions/runs/14571295945/job/40868936348#step:7:33
author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-21T13:08:03Z
body: Updating the branch to get the macOS artifacts built.

What is the risk this causes us to hit the iOS memory ceiling?
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-21T13:09:42Z
body: > Updating the branch to get the macOS artifacts built.

I already rebased and rebuilt it. You can find them here: https://github.com/firezone/firezone/actions/runs/14573525934/job/40874959638
author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-21T13:11:09Z
body: > > Updating the branch to get the macOS artifacts built.
> 
> I already rebased and rebuilt it. You can find them here: https://github.com/firezone/firezone/actions/runs/14573525934/job/40874959638

Oh, sorry. Feel free to force-push again to remove that commit if desired.
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-21T13:11:10Z
body: > What is the risk this causes us to hit the iOS memory ceiling?

I don't think kernel buffers count towards the application memory limit.
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-21T13:12:40Z
body: > Is this for each UDP socket opened by the IPC service? I'll probably spend some time measuring memory usage increase.

Yes it is for each UDP socket we open. That also includes the ones we create for making DNS queries.

> Would that be expected to grow with the number of Gateways a user connects to?

No, we use the same UDP socket for all Gateways & Relay communication.
author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-21T14:01:37Z
body: The apps sent for review last night are ready without this PR, so I'll go ahead and do a cut of those while I continue to test this. Just want to be extra sure this is safe (it's a big jump from what the sizes are now).
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-21T21:11:11Z
body: > The apps sent for review last night are ready without this PR, so I'll go ahead and do a cut of those while I continue to test this. Just want to be extra sure this is safe (it's a big jump from what the sizes are now).

Sounds good to me!
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-22T00:21:55Z
body: @jamilbk Did you see any throughput improvements with this PR? Because if not, then we don't need to bother increasing the buffer sizes on MacOS and can save ourselves the memory testing (although I think it would be good to confirm that it does in fact no affect user-space RAM usage).

Edit: Built a new client here: https://github.com/firezone/firezone/actions/runs/14584121808/job/40906492534
author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-22T00:59:30Z
body: I wasn't able to measure any immediately obvious throughput improvements on macOS. There may be a slight increase in the downstream path. I'm trying to test on iOS now. I need to build the swift workflow for this branch to get a release TestFlight client.

At least on macOS I _may_ have seen an additional 2-3 MB of memory usage (up to ~38 MB) but it's hard to say if that's just due to more DNS requests that happened to be made.

38 MB is (IMO) getting dangerously close to the iOS 50 MB hard limit, so I'm doing more testing now.

Regardless of this PR or not, something in the upstream path in Apple is still slowing things down significantly with added latency in the upstream path.

Let me print the _actual_ socket size the OS gives us because I think it's either clamping here, or we're hitting another small buffer in the upstream path for Apple (utun?).

# macOS before

<img width="3120" alt="Screenshot 2025-04-21 at 5 57 07 PM" src="https://github.com/user-attachments/assets/a71bfbfe-b06a-4a75-82a4-e54b93ad8753" />


# macOS with this PR

<img width="3120" alt="Screenshot 2025-04-21 at 5 35 50 PM" src="https://github.com/user-attachments/assets/9c916f63-f036-42a8-a174-354998c5570f" />

author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-22T01:14:21Z
body: This might explain the download / upload discrepancy:

```
2025-04-21 18:13:11.031003 -0700	dev.firezone.firezone.network-extension	firezone_tunnel::sockets: Bound new IPv4 UDP p2p socket send_buf_size=1048576 recv_buf_size=8388608 port=49235
2025-04-21 18:13:11.031052 -0700	dev.firezone.firezone.network-extension	firezone_tunnel::sockets: Bound new IPv6 UDP p2p socket send_buf_size=1048576 recv_buf_size=8388608 port=63765
```
author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-22T01:16:55Z
body: 1 MB for 100 ms latency should still allow around 80 Mbps. I definitely don't see a +9 MB increase in userspace memory usage in the network extension process, so it's probably safe to say the buffer size only affects kernel memory (but on iOS, I'm not sure if that factors into our allowance even though it's kernel space).
author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-22T01:22:59Z
body: @thomaseizinger Thoughts on trying 8 MB for send *and* recv on Apple?
author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-22T01:25:35Z
body: What might be smart to do here is use much smaller buffers for the DNS queries since those don't need much bandwidth. The WireGuard socket will be the one where we need the 10 MB.
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-22T02:03:59Z
body: > @thomaseizinger Thoughts on trying 8 MB for send _and_ recv on Apple?

You are welcome to experiment with it (it is just a small code change). As argued in the PR description, I think there are diminishing returns in increasing the send-buffer size.
author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-22T02:25:19Z
body: > > @thomaseizinger Thoughts on trying 8 MB for send _and_ recv on Apple?
> 
> 
> 
> You are welcome to experiment with it (it is just a small code change). As argued in the PR description, I think there are diminishing returns in increasing the send-buffer size.

I'll push up another branch with a test.

I disagree about the assumption that upstream is not as critical. For consumer use cases yes (browsing, streaming video) but I think in corporate settings the expectation is the speeds will roughly track line rate minus some overhead to be roughly in line with the vpn they were using before.

Also the default iperf3 test many customers seem to be running defaults to testing upload :-/.
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-22T02:31:15Z
body: > I disagree about the assumption that upstream is not as critical. For consumer use cases yes (browsing, streaming video) but I think in corporate settings the expectation is the speeds will roughly track line rate minus some overhead to be roughly in line with the vpn they were using before.
> 
> Also the default iperf3 test many customers seem to be running defaults to testing upload :-/.

We are disagreeing here on what we are debating. I am not saying upload is not critical. I am saying send buffer sizes are not as much of a critical component when it comes to achieving high throughput. `connlib` will automatically suspend writing when the send buffer is full. This will cause datagrams within our internal queues to build up to a point where those are full at which point we will stop reading from the TUN device, causing either packets to be dropped there or the sending application to be back-pressured with `EWOULDBLOCK` due to full buffers.

The kernel will already flush out UDP packets as fast as it can from the buffer to the wire. Increasing this buffer size is not going to make that any faster. My understanding of how this works together is that we have back-pressure with extremely low latency to the producer and therefore don't need large buffers.

The same is not true over the network. Receive buffers need to larger with increased latency because the remote needs to ackknowledge that it received a certain packet in order for backpressure to work.
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-22T02:33:43Z
body: > I wasn't able to measure any immediately obvious throughput improvements on macOS. There may be a slight increase in the downstream path. I'm trying to test on iOS now. I need to build the swift workflow for this branch to get a release TestFlight client.
> 
> At least on macOS I _may_ have seen an additional 2-3 MB of memory usage (up to ~38 MB) but it's hard to say if that's just due to more DNS requests that happened to be made.
> 
> 38 MB is (IMO) getting dangerously close to the iOS 50 MB hard limit, so I'm doing more testing now.
> 
> Regardless of this PR or not, something in the upstream path in Apple is still slowing things down significantly with added latency in the upstream path.
> 
> Let me print the _actual_ socket size the OS gives us because I think it's either clamping here, or we're hitting another small buffer in the upstream path for Apple (utun?).
> # macOS before
> <img alt="Screenshot 2025-04-21 at 5 57 07 PM" width="2000" src="https://private-user-images.githubusercontent.com/167144/435858821-a71bfbfe-b06a-4a75-82a4-e54b93ad8753.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDUyODY1MzcsIm5iZiI6MTc0NTI4NjIzNywicGF0aCI6Ii8xNjcxNDQvNDM1ODU4ODIxLWE3MWJmYmZlLWIwNmEtNGE3NS04MmE0LWU1NGI5M2FkODc1My5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwNDIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDQyMlQwMTQzNTdaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT05NDY2MmEwZjgzNTYyNjY0ZjFhYThhMmYzNWEzNTM4Y2VhY2I5YzBhOGExNjgzMDRjZWIzMjEwN2NiMzNhZjFmJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.EDF1uSXJyowuC8Y2dN6nTVTKw52J3_FckUcjFuB1Sy8">
> # macOS with this PR
> <img alt="Screenshot 2025-04-21 at 5 35 50 PM" width="2000" src="https://private-user-images.githubusercontent.com/167144/435858907-9c916f63-f036-42a8-a174-354998c5570f.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NDUyODY1MzcsIm5iZiI6MTc0NTI4NjIzNywicGF0aCI6Ii8xNjcxNDQvNDM1ODU4OTA3LTljOTE2ZjYzLWYwMzYtNDJhOC1hMTc0LTM1NDk5OGM1NTcwZi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjUwNDIyJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI1MDQyMlQwMTQzNTdaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xOWQ5ZGNiOGMxNjFlODdkYmRiZDdlZWM5ZDliNmUxMDhhMTgwYjU5ZjVhOTJhYjcxNTI0YWVjOWY3MWU0N2NjJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCJ9.UHPPa3p7tAYMEJLob4CONYT2DVOZ-9RlcyK7igyzFF0">

These are awesome results! That is a 30% increase in download speed!

My guess would be the increased memory usage comes from the additional threads that we have been adding.
author_association: MEMBER
user: login: thomaseizinger
created_at: 2025-04-22T02:34:55Z
body: > What might be smart to do here is use much smaller buffers for the DNS queries since those don't need much bandwidth. The WireGuard socket will be the one where we need the 10 MB.

Moved the setting of increased buffers to a setter that we only call for the P2P sockets.
author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-22T02:57:16Z
body: > > I disagree about the assumption that upstream is not as critical. For consumer use cases yes (browsing, streaming video) but I think in corporate settings the expectation is the speeds will roughly track line rate minus some overhead to be roughly in line with the vpn they were using before.
> > Also the default iperf3 test many customers seem to be running defaults to testing upload :-/.
> 
> We are disagreeing here on what we are debating. I am not saying upload is not critical. I am saying send buffer sizes are not as much of a critical component when it comes to achieving high throughput. `connlib` will automatically suspend writing when the send buffer is full. This will cause datagrams within our internal queues to build up to a point where those are full at which point we will stop reading from the TUN device, causing either packets to be dropped there or the sending application to be back-pressured with `EWOULDBLOCK` due to full buffers.
> 
> The kernel will already flush out UDP packets as fast as it can from the buffer to the wire. Increasing this buffer size is not going to make that any faster. My understanding of how this works together is that we have back-pressure with extremely low latency to the producer and therefore don't need large buffers.
> 
> The same is not true over the network. Receive buffers need to larger with increased latency because the remote needs to ackknowledge that it received a certain packet in order for backpressure to work.

I knew I was probably missing something, makes sense.
author_association: MEMBER
user: login: jamilbk
created_at: 2025-04-22T02:58:04Z
body: > These are awesome results! That is a 30% increase in download speed!

Hm maybe, bear in mind the Gateway is on a burstable CPU so YMMV. I think it's probably safe to merge and at least rules one of the possible buffer size constraints out.
author_association: MEMBER