Repository: helidon-io/helidon (java)

Issue:
4.0 webserver extremely slow on linux
Running Helidon 4.0 on Linux shows really slow performance. **The exact code runs on OSX as expected.**

The startup shows:

```
Nov 09, 2023 4:44:04 PM io.helidon.webserver.ServerListener start
INFO: [0x34e21d23] http://127.0.0.1:8888 bound for socket '@default'
Nov 09, 2023 4:44:04 PM io.helidon.common.features.HelidonFeatures features
INFO: Helidon SE 4.0.0 features: [Config, Encoding, Media, WebServer]
Nov 09, 2023 4:44:04 PM io.helidon.webserver.LoomServer startIt
INFO: Started all channels in 60 milliseconds. 289 milliseconds since JVM startup. Java 21+35-LTS-2513
```

The Linux environment is:

`5.15.0-88-generic #98~20.04.1-Ubuntu SMP Mon Oct 9 16:43:45 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux`

You can open the Helidon test project [here](https://github.com/robaho/helidon_test)

There are instructions on using h2load in the readme.

The "baseline" for the test is available [here](https://github.com/robaho/enobufs_test).

Running against the enobufs baseline:

```
rengels:~$ h2load -n100000 -c4 -m10 -t4 --h1 http://localhost:8888/devnull?size=1000000
starting benchmark...
spawning thread #0: 1 total client(s). 25000 total requests
spawning thread #1: 1 total client(s). 25000 total requests
spawning thread #2: 1 total client(s). 25000 total requests
spawning thread #3: 1 total client(s). 25000 total requests
Application protocol: http/1.1
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 15.68s, 6375.89 req/s, 5.94GB/s
requests: 100000 total, 100000 started, 100000 done, 100000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 100000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 93.14GB (100006800000) total, 3.91MB (4100000) headers (space savings 0.00%), 93.13GB (100000000000) data
                     min         max         mean         sd        +/- sd
time for request:     3.26ms     34.84ms      6.24ms      1.22ms    81.98%
time for connect:      125us       201us       149us        34us    75.00%
time to 1st byte:    15.66ms     17.08ms     16.19ms       627us    75.00%
req/s           :    1594.00     1609.09     1601.37        7.05    50.00%
rengels:~$ 
```

I can't even run the same the test against Helidon since it takes too long. **Lowering the number of requests from 100k to 1k (it seems to be an O(N^2) increase, I get:**

```
rengels:~$ h2load -n1000 -c4 -m10 -t4 --h1 http://localhost:8888/devnull?size=1000000
starting benchmark...
spawning thread #0: 1 total client(s). 250 total requests
spawning thread #1: 1 total client(s). 250 total requests
spawning thread #2: 1 total client(s). 250 total requests
spawning thread #3: 1 total client(s). 250 total requests
Application protocol: http/1.1
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 165.26s, 6.05 req/s, 5.77MB/s
requests: 1000 total, 1000 started, 1000 done, 1000 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 1000 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 953.81MB (1000146000) total, 108.40KB (111000) headers (space savings 0.00%), 953.67MB (1000000000) data
                     min         max         mean         sd        +/- sd
time for request:     2.47ms       6.73s       6.48s    866.25ms    96.20%
time for connect:      175us       211us       193us        15us    50.00%
time to 1st byte:     1.53ms      1.79ms      1.68ms       120us    75.00%
req/s           :       1.51        1.52        1.51        0.00    75.00%

```

The code that prompted this investigation is [here](https://github.com/robaho/httpserver) which uses a similar design as Helidon and runs the same on Linux and OSX and achieves the same performance as the baseline.

Like I started with, the Helidon code runs as expected when running the test on OSX.





Pull request:
4.x: Re-enable TCP auto-tuning
Fixes #7983


Patch:
diff --git a/common/socket/src/main/java/io/helidon/common/socket/SocketOptionsBlueprint.java b/common/socket/src/main/java/io/helidon/common/socket/SocketOptionsBlueprint.java
index c487c778a2..4fc8d864eb 100644
--- a/common/socket/src/main/java/io/helidon/common/socket/SocketOptionsBlueprint.java
+++ b/common/socket/src/main/java/io/helidon/common/socket/SocketOptionsBlueprint.java
@@ -23,6 +23,7 @@ import java.net.SocketOption;
 import java.net.StandardSocketOptions;
 import java.time.Duration;
 import java.util.Map;
+import java.util.Optional;
 
 import io.helidon.builder.api.Option;
 import io.helidon.builder.api.Prototype;
@@ -67,8 +68,8 @@ interface SocketOptionsBlueprint {
      * @return buffer size, in bytes
      * @see java.net.StandardSocketOptions#SO_RCVBUF
      */
-    @ConfiguredOption("32768")
-    int socketReceiveBufferSize();
+    @ConfiguredOption
+    Optional<Integer> socketReceiveBufferSize();
 
     /**
      * Socket send buffer size.
@@ -76,8 +77,8 @@ interface SocketOptionsBlueprint {
      * @return buffer size, in bytes
      * @see java.net.StandardSocketOptions#SO_SNDBUF
      */
-    @ConfiguredOption("32768")
-    int socketSendBufferSize();
+    @ConfiguredOption
+    Optional<Integer> socketSendBufferSize();
 
     /**
      * Socket reuse address.
@@ -129,8 +130,8 @@ interface SocketOptionsBlueprint {
     class BuilderDecorator implements Prototype.BuilderDecorator<SocketOptions.BuilderBase<?, ?>> {
         @Override
         public void decorate(SocketOptions.BuilderBase<?, ?> target) {
-            target.putSocketOption(StandardSocketOptions.SO_RCVBUF, target.socketReceiveBufferSize());
-            target.putSocketOption(StandardSocketOptions.SO_SNDBUF, target.socketSendBufferSize());
+            target.socketReceiveBufferSize().ifPresent(i -> target.putSocketOption(StandardSocketOptions.SO_RCVBUF, i));
+            target.socketSendBufferSize().ifPresent(i -> target.putSocketOption(StandardSocketOptions.SO_SNDBUF, i));
             target.putSocketOption(StandardSocketOptions.SO_REUSEADDR, target.socketReuseAddress());
             target.putSocketOption(StandardSocketOptions.SO_KEEPALIVE, target.socketKeepAlive());
             target.putSocketOption(StandardSocketOptions.TCP_NODELAY, target.tcpNoDelay());
diff --git a/metrics/providers/micrometer/src/main/java/io/helidon/metrics/providers/micrometer/MicrometerPrometheusFormatter.java b/metrics/providers/micrometer/src/main/java/io/helidon/metrics/providers/micrometer/MicrometerPrometheusFormatter.java
index ebad78514a..c8aa27050f 100644
--- a/metrics/providers/micrometer/src/main/java/io/helidon/metrics/providers/micrometer/MicrometerPrometheusFormatter.java
+++ b/metrics/providers/micrometer/src/main/java/io/helidon/metrics/providers/micrometer/MicrometerPrometheusFormatter.java
@@ -131,10 +131,10 @@ public class MicrometerPrometheusFormatter implements MeterRegistryFormatter {
                 return Optional.empty();
             }
 
-            String prometheusOutput = filter(prometheusMeterRegistry.get()
-                                                     .scrape(MicrometerPrometheusFormatter.MEDIA_TYPE_TO_FORMAT.get(
-                                                                     resultMediaType),
-                                                             meterNamesOfInterest));
+            String prometheusOutput = prometheusMeterRegistry.get()
+                    .scrape(MicrometerPrometheusFormatter.MEDIA_TYPE_TO_FORMAT.get(
+                                    resultMediaType),
+                            meterNamesOfInterest);
 
             return prometheusOutput.isBlank() ? Optional.empty() : Optional.of(prometheusOutput);
         }
@@ -218,16 +218,6 @@ public class MicrometerPrometheusFormatter implements MeterRegistryFormatter {
         return result;
     }
 
-    /**
-     * Filter the Prometheus-format report.
-     *
-     * @param output Prometheus-format report
-     * @return output filtered
-     */
-    private static String filter(String output) {
-        return output.replaceFirst("# EOF\r?\n?", "");
-    }
-
     private static Optional<PrometheusMeterRegistry> prometheusMeterRegistry(MeterRegistry meterRegistry) {
         io.micrometer.core.instrument.MeterRegistry mMeterRegistry =
                 meterRegistry.unwrap(io.micrometer.core.instrument.MeterRegistry.class);


Test patch:
diff --git a/metrics/providers/micrometer/src/test/java/io/helidon/metrics/providers/micrometer/TestPrometheusFormatting.java b/metrics/providers/micrometer/src/test/java/io/helidon/metrics/providers/micrometer/TestPrometheusFormatting.java
index ac00f6e0c1..10a98c3103 100644
--- a/metrics/providers/micrometer/src/test/java/io/helidon/metrics/providers/micrometer/TestPrometheusFormatting.java
+++ b/metrics/providers/micrometer/src/test/java/io/helidon/metrics/providers/micrometer/TestPrometheusFormatting.java
@@ -19,6 +19,7 @@ import java.util.Optional;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 
+import io.helidon.common.media.type.MediaTypes;
 import io.helidon.common.testing.junit5.OptionalMatcher;
 import io.helidon.metrics.api.Counter;
 import io.helidon.metrics.api.MeterRegistry;
@@ -36,6 +37,7 @@ import org.junit.jupiter.api.Test;
 import static org.hamcrest.MatcherAssert.assertThat;
 import static org.hamcrest.Matchers.allOf;
 import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.endsWith;
 import static org.hamcrest.Matchers.instanceOf;
 import static org.hamcrest.Matchers.is;
 import static org.hamcrest.Matchers.not;
@@ -43,6 +45,13 @@ import static org.hamcrest.Matchers.not;
 class TestPrometheusFormatting {
 
     private static final String SCOPE_TAG_NAME = "this-scope";
+
+    /*
+    Only OpenMetrics format, not the Prometheus exposition format, has the trailing EOF which (for example) the Prometheus
+    server expects to see when it probes targets. That's why, below, when we format the output, we specify OpenMetrics as the
+    media type.
+     */
+    private static final String OPENMETRICS_EOF = "# EOF\n";
     private static MeterRegistry meterRegistry;
 
     private static MetricsConfig metricsConfig;
@@ -87,13 +96,15 @@ class TestPrometheusFormatting {
         e.record(2, TimeUnit.SECONDS);
 
         MicrometerPrometheusFormatter formatter = MicrometerPrometheusFormatter.builder(meterRegistry)
+                .resultMediaType(MediaTypes.APPLICATION_OPENMETRICS_TEXT)
                 .scopeTagName(SCOPE_TAG_NAME)
                 .build();
         Optional<Object> outputOpt = formatter.format();
 
         assertThat("Formatted output",
                    checkAndCast(outputOpt),
-                   allOf(containsString(scopeExpr("c1_total",
+                   allOf(
+                           containsString(scopeExpr("c1_total",
                                                   "this_scope",
                                                   "app",
                                                   "1.0")),
@@ -108,7 +119,9 @@ class TestPrometheusFormatting {
                          containsString(scopeExpr("t1_1_seconds_count",
                                                   "this_scope",
                                                   "app",
-                                                  "1.0"))));
+                                                  "1.0")),
+                         endsWith(OPENMETRICS_EOF)));
+
     }
 
     @Test
@@ -122,6 +135,7 @@ class TestPrometheusFormatting {
         d.record(7, TimeUnit.SECONDS);
 
         MicrometerPrometheusFormatter formatter = MicrometerPrometheusFormatter.builder(meterRegistry)
+                .resultMediaType(MediaTypes.APPLICATION_OPENMETRICS_TEXT)
                 .scopeTagName(SCOPE_TAG_NAME)
                 .meterNameSelection(Set.of("c2"))
                 .build();
@@ -139,7 +153,8 @@ class TestPrometheusFormatting {
                                                       "1.0"))),
                          not(containsString(scopeExpr("t2_seconds_sum",
                                                       "this_scope",
-                                                      "app", "7.0")))));
+                                                      "app", "7.0"))),
+                         endsWith(OPENMETRICS_EOF)));
 
     }
 
@@ -159,6 +174,7 @@ class TestPrometheusFormatting {
         e.record(2, TimeUnit.SECONDS);
 
         MicrometerPrometheusFormatter formatter = MicrometerPrometheusFormatter.builder(meterRegistry)
+                .resultMediaType(MediaTypes.APPLICATION_OPENMETRICS_TEXT)
                 .scopeTagName(SCOPE_TAG_NAME)
                 .scopeSelection(Set.of("app"))
                 .build();
@@ -182,11 +198,12 @@ class TestPrometheusFormatting {
                          containsString(scopeExpr("t3_1_seconds_count",
                                                   "this_scope",
                                                   "app",
-                                                  "1.0"))));
+                                                  "1.0")),
+                         endsWith(OPENMETRICS_EOF)));
     }
 
     private static String scopeExpr(String meterName, String key, String value, String suffix) {
-        return meterName + "{" + key + "=\"" + value + "\",} " + suffix;
+        return meterName + "{" + key + "=\"" + value + "\"} " + suffix;
     }
 
     private static String checkAndCast(Optional<Object> outputOpt) {