Repository: helidon-io/helidon (java)

Issue:
[Helidon 4 SE] Metrics format not compatible with Prometheus: data does not end with # EOF
[//]: # "Please remove these comments"
[//]: # "Provide a general summary of the issue in the Title above"

## Environment Details
* Helidon Version: 4
* Helidon SE
* JDK version: 21
* OS: macOs Ventura 13.6.1
* Docker version: 20.10.17
* Prometheus version (Docker): prom/prometheus:v2.4.72

----------

## Problem Description
I'm not able to configure Prometheus to ingest the metrics exposed at the `/observe/metrics endpoint` as described in [this guide](https://helidon.io/docs/v4/#/se/guides/metrics).
The error I'm getting on Prometheus when trying to scrape the endpoint is `data does not end with # EOF`, which seems similar to what is described on issue #4884 for Helidon 2.x and 3.x.

![Screenshot 2023-11-09 at 10 22 28](https://github.com/helidon-io/helidon/assets/10221348/77fbc549-d1a4-4b32-b37b-fb04de204dd7)


## Steps to reproduce
- Follow [this guide](https://helidon.io/docs/v4/#/se/guides/metrics) and register the default metrics.
- Spin up a Prometheus docker container and add the `/observe/metrics` endpoint as target.
- Check the target status on `${prometheusHost:Port}/targets`



Pull request description:
Keep EOF at end of OpenMetrics output
### Description
Resolves #7981

Now that Helidon's metrics is based on Micrometer, we use Micrometer's Prometheus meter registry for formatting the output. The Prometheus meter registry handles both Prometheus exposition format (which does not include a trailing EOF line) and OpenMetrics (which does).

That said, our formatting needs to do some processing on the output from that registry before returning it and that processing was incorrectly removing the `# EOF` string from the end of OpenMetrics output.

The changes here remove that trimming and also enhance unit tests of the formatter to:
1. Select OpenMetrics as the output (since that's the format, not the Prometheus exposition format, that has the `# EOF` trailer.
2. Make sure the formatted output ends with that trailer.
3. Slightly revise the expected results. Prometheus format allows for (and the Micrometer Prometheus meter registry includes) a trailing `,` after tags; the OpenMetrics output does not have that trailing `,` and since the tests now specify OpenMetrics format they need to _not_ expect the trailing `,` after tags.

As a test specific to the use case in the issue, I ran the SE Quickstart app and started a Prometheus server locally using the `prometheus.yml` configuration file below. The Prometheus server now accepts and processes the response from the metrics endpoint. (Most of the config is boilerplate; the interesting part is the last `job` in the `scrape_config` section.)

```yaml
# my global config
global:
  scrape_interval:     5s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

  - job_name: 'helidon'
    metrics_path: /observe/metrics
    static_configs:
    - targets: ['localhost:8080']
```

### Documentation
No doc impact - this is a bug fix.


Patch:
diff --git a/archetypes/helidon/src/main/archetype/common/docker.xml b/archetypes/helidon/src/main/archetype/common/docker.xml
index a09bbe76cc..5ed2e9376c 100644
--- a/archetypes/helidon/src/main/archetype/common/docker.xml
+++ b/archetypes/helidon/src/main/archetype/common/docker.xml
@@ -62,11 +62,11 @@
                     </includes>
                 </templates>
                 <model>
-                    <list key="poms">
-                        <value>pom.xml</value>
+                    <list key="poms" if="!${multi-module}">
+                        <value>ADD pom.xml .</value>
                     </list>
                     <list key="src-dirs" if="!${multi-module}">
-                        <value>src</value>
+                        <value>ADD src src</value>
                     </list>
                     <list key="native-sections" if="${docker.native-image}">
                         <value file="files/README.native.md"/>
diff --git a/archetypes/helidon/src/main/archetype/common/files/Dockerfile.jlink.mustache b/archetypes/helidon/src/main/archetype/common/files/Dockerfile.jlink.mustache
index df76d3b74d..2b7f84508e 100644
--- a/archetypes/helidon/src/main/archetype/common/files/Dockerfile.jlink.mustache
+++ b/archetypes/helidon/src/main/archetype/common/files/Dockerfile.jlink.mustache
@@ -1,6 +1,16 @@
 
 # 1st stage, build the app
-FROM maven:3.8.4-openjdk-17-slim as build
+FROM container-registry.oracle.com/java/openjdk:21 as build
+
+WORKDIR /usr/share
+
+# Install maven
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
 
 WORKDIR /helidon
 
diff --git a/archetypes/helidon/src/main/archetype/common/files/Dockerfile.mustache b/archetypes/helidon/src/main/archetype/common/files/Dockerfile.mustache
index 73d1f16553..883d9dfade 100644
--- a/archetypes/helidon/src/main/archetype/common/files/Dockerfile.mustache
+++ b/archetypes/helidon/src/main/archetype/common/files/Dockerfile.mustache
@@ -17,14 +17,14 @@ WORKDIR /helidon
 # Incremental docker builds will always resume after that, unless you update
 # the pom
 {{#poms}}
-ADD {{.}} {{.}}
+{{.}}
 {{/poms}}
 RUN mvn package -Dmaven.test.skip -Declipselink.weave.skip {{#docker-phase1-options}}{{.}}{{^last}} {{/last}}{{/docker-phase1-options}}
 
 # Do the Maven build!
 # Incremental docker builds will resume here when you change sources
 {{#src-dirs}}
-ADD {{.}} {{.}}
+{{.}}
 {{/src-dirs}}
 RUN mvn package -DskipTests
 
diff --git a/archetypes/helidon/src/main/archetype/common/files/pom.xml.mustache b/archetypes/helidon/src/main/archetype/common/files/pom.xml.mustache
index cba530bef0..d690036acf 100644
--- a/archetypes/helidon/src/main/archetype/common/files/pom.xml.mustache
+++ b/archetypes/helidon/src/main/archetype/common/files/pom.xml.mustache
@@ -15,11 +15,11 @@
     <artifactId>{{artifactId}}</artifactId>
     <version>{{project-version}}</version>
 
-    <properties>
 {{#mainClass}}
+    <properties>
         <mainClass>{{mainClass}}</mainClass>
-{{/mainClass}}
     </properties>
+{{/mainClass}}
 
     <dependencies>
         {{#dependencies}}
diff --git a/archetypes/helidon/src/main/archetype/mp/oci/oci-mp.xml b/archetypes/helidon/src/main/archetype/mp/oci/oci-mp.xml
index 31ce157a21..afdb6cf6d7 100644
--- a/archetypes/helidon/src/main/archetype/mp/oci/oci-mp.xml
+++ b/archetypes/helidon/src/main/archetype/mp/oci/oci-mp.xml
@@ -397,13 +397,14 @@ Mount this volume as part of your application containers specification
                                 ]]></value>
             </list>
             <list key="poms">
-                <value>client/pom.xml</value>
-                <value>server/pom.xml</value>
+                <value>ADD pom.xml pom.xml</value>
+                <value>ADD client/pom.xml client/pom.xml</value>
+                <value>ADD server/pom.xml server/pom.xml</value>
             </list>
             <list key="src-dirs">
-                <value>client/src</value>
-                <value>server/src</value>
-                <value>spec/api.yaml</value>
+                <value>ADD client/src client/src</value>
+                <value>ADD server/src server/src</value>
+                <value>ADD spec/api.yaml spec/api.yaml</value>
             </list>
         </model>
     </output>
diff --git a/docs/mp/guides/mp-tutorial.adoc b/docs/mp/guides/mp-tutorial.adoc
index 6fb22815d0..3e5fa20b05 100644
--- a/docs/mp/guides/mp-tutorial.adoc
+++ b/docs/mp/guides/mp-tutorial.adoc
@@ -948,7 +948,17 @@ Add a new `Dockerfile` in the project root directory with the following content:
 [source,bash]
 .Dockerfile content
 ----
-FROM maven:3.8.4-openjdk-17-slim as build <1>
+FROM container-registry.oracle.com/java/openjdk:21 as build <1>
+
+# Install maven
+WORKDIR /usr/share
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
+
 WORKDIR /helidon
 
 ADD pom.xml .
@@ -958,13 +968,13 @@ ADD src src
 RUN mvn package -DskipTests <3>
 RUN echo "done!"
 
-FROM openjdk:17-jdk-slim <4>
+FROM container-registry.oracle.com/java/openjdk:21
 WORKDIR /helidon
 
-COPY --from=build /helidon/target/helidon-mp-tutorial.jar ./ <5>
+COPY --from=build /helidon/target/helidon-mp-tutorial.jar ./ <4>
 COPY --from=build /helidon/target/libs ./libs
 
-CMD ["java", "-jar", "helidon-mp-tutorial.jar"] <6>
+CMD ["java", "-jar", "helidon-mp-tutorial.jar"] <5>
 EXPOSE 8080
 ----
 
@@ -977,9 +987,8 @@ EXPOSE 8080
  builds faster because they will use this cached layer rather than downloading
  everything again.
 <3> Add the source code and do the real build.
-<4> Start a second stage using a much smaller runtime image.
-<5> Copy the binary and libraries from the first stage.
-<6> Set the initial command and expose port 8080.
+<4> Copy the binary and libraries from the first stage.
+<5> Set the initial command and expose port 8080.
 
 To create the Docker image, use the following command:
 
diff --git a/examples/employee-app/Dockerfile b/examples/employee-app/Dockerfile
index 29b4c38700..17710481a7 100644
--- a/examples/employee-app/Dockerfile
+++ b/examples/employee-app/Dockerfile
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2019, 2021 Oracle and/or its affiliates.
+# Copyright (c) 2019, 2023 Oracle and/or its affiliates.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,16 @@
 #
 
 # 1st stage, build the app
-FROM maven:3.6.3-openjdk-17-slim as build
+FROM container-registry.oracle.com/java/openjdk:21 as build
+
+# Install maven
+WORKDIR /usr/share
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
 
 WORKDIR /helidon
 
@@ -33,7 +42,7 @@ RUN mvn package -DskipTests
 RUN echo "done!"
 
 # 2nd stage, build the runtime image
-FROM openjdk:17-jdk-slim
+FROM container-registry.oracle.com/java/openjdk:21
 WORKDIR /helidon
 
 # Copy the binary built in the 1st stage
diff --git a/examples/integrations/cdi/datasource-hikaricp/Dockerfile b/examples/integrations/cdi/datasource-hikaricp/Dockerfile
index 13dcb075d7..de49c098f2 100644
--- a/examples/integrations/cdi/datasource-hikaricp/Dockerfile
+++ b/examples/integrations/cdi/datasource-hikaricp/Dockerfile
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2018, 2021 Oracle and/or its affiliates.
+# Copyright (c) 2018, 2023 Oracle and/or its affiliates.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,16 @@
 #
 
 # 1st stage, build the app
-FROM maven:3.6.3-openjdk-17-slim as build
+FROM container-registry.oracle.com/java/openjdk:21 as build
+
+# Install maven
+WORKDIR /usr/share
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
 
 WORKDIR /helidon
 
@@ -32,7 +41,7 @@ RUN mvn package -DskipTests
 RUN echo "done!"
 
 # 2nd stage, build the runtime image
-FROM openjdk:17-jdk-slim
+FROM container-registry.oracle.com/java/openjdk:21
 WORKDIR /helidon
 
 # Copy the binary built in the 1st stage
diff --git a/examples/quickstarts/helidon-quickstart-mp/Dockerfile b/examples/quickstarts/helidon-quickstart-mp/Dockerfile
index e7a36c5082..d2466ddbac 100644
--- a/examples/quickstarts/helidon-quickstart-mp/Dockerfile
+++ b/examples/quickstarts/helidon-quickstart-mp/Dockerfile
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2018, 2022 Oracle and/or its affiliates.
+# Copyright (c) 2018, 2023 Oracle and/or its affiliates.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,16 @@
 #
 
 # 1st stage, build the app
-FROM maven:3.8.4-openjdk-17-slim as build
+FROM container-registry.oracle.com/java/openjdk:21 as build
+
+# Install maven
+WORKDIR /usr/share
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
 
 WORKDIR /helidon
 
@@ -32,7 +41,7 @@ RUN mvn package -DskipTests
 RUN echo "done!"
 
 # 2nd stage, build the runtime image
-FROM openjdk:17-jdk-slim
+FROM container-registry.oracle.com/java/openjdk:21
 WORKDIR /helidon
 
 # Copy the binary built in the 1st stage
diff --git a/examples/quickstarts/helidon-quickstart-mp/Dockerfile.jlink b/examples/quickstarts/helidon-quickstart-mp/Dockerfile.jlink
index 1f0bc842fd..cc482519f6 100644
--- a/examples/quickstarts/helidon-quickstart-mp/Dockerfile.jlink
+++ b/examples/quickstarts/helidon-quickstart-mp/Dockerfile.jlink
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2020, 2022 Oracle and/or its affiliates.
+# Copyright (c) 2020, 2023 Oracle and/or its affiliates.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,16 @@
 #
 
 # 1st stage, build the app
-FROM maven:3.8.4-openjdk-17-slim as build
+FROM container-registry.oracle.com/java/openjdk:21 as build
+
+# Install maven
+WORKDIR /usr/share
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
 
 WORKDIR /helidon
 
diff --git a/examples/quickstarts/helidon-quickstart-se/Dockerfile.jlink b/examples/quickstarts/helidon-quickstart-se/Dockerfile.jlink
index 7906e826c2..245e889702 100644
--- a/examples/quickstarts/helidon-quickstart-se/Dockerfile.jlink
+++ b/examples/quickstarts/helidon-quickstart-se/Dockerfile.jlink
@@ -15,7 +15,7 @@
 #
 
 # 1st stage, build the app
-FROM container-registry.oracle.com/java/openjdk:21 as maven
+FROM container-registry.oracle.com/java/openjdk:21 as build
 
 # Install maven
 WORKDIR /usr/share
diff --git a/examples/quickstarts/helidon-standalone-quickstart-mp/Dockerfile b/examples/quickstarts/helidon-standalone-quickstart-mp/Dockerfile
index b3ca729710..0e6c1e2c6b 100644
--- a/examples/quickstarts/helidon-standalone-quickstart-mp/Dockerfile
+++ b/examples/quickstarts/helidon-standalone-quickstart-mp/Dockerfile
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2018, 2022 Oracle and/or its affiliates.
+# Copyright (c) 2018, 2023 Oracle and/or its affiliates.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,16 @@
 #
 
 # 1st stage, build the app
-FROM maven:3.8.4-openjdk-17-slim as build
+FROM container-registry.oracle.com/java/openjdk:21 as build
+
+# Install maven
+WORKDIR /usr/share
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
 
 WORKDIR /helidon
 
@@ -32,7 +41,7 @@ RUN mvn package -DskipTests
 RUN echo "done!"
 
 # 2nd stage, build the runtime image
-FROM openjdk:17-jdk-slim
+FROM container-registry.oracle.com/java/openjdk:21
 WORKDIR /helidon
 
 # Copy the binary built in the 1st stage
diff --git a/examples/quickstarts/helidon-standalone-quickstart-mp/Dockerfile.jlink b/examples/quickstarts/helidon-standalone-quickstart-mp/Dockerfile.jlink
index 1f0bc842fd..c76160bcae 100644
--- a/examples/quickstarts/helidon-standalone-quickstart-mp/Dockerfile.jlink
+++ b/examples/quickstarts/helidon-standalone-quickstart-mp/Dockerfile.jlink
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2020, 2022 Oracle and/or its affiliates.
+# Copyright (c) 2020, 2023 Oracle and/or its affiliates.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,16 @@
 #
 
 # 1st stage, build the app
-FROM maven:3.8.4-openjdk-17-slim as build
+FROM container-registry.oracle.com/java/openjdk:21 as build
+
+# Install maven
+WORKDIR /usr/share
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
 
 WORKDIR /helidon
 
@@ -35,6 +44,6 @@ RUN echo "done!"
 
 FROM debian:stretch-slim
 WORKDIR /helidon
-COPY --from=build /helidon/target/helidon-quickstart-mp-jri ./
+COPY --from=build /helidon/target/helidon-standalone-quickstart-mp-jri ./
 ENTRYPOINT ["/bin/bash", "/helidon/bin/start"]
 EXPOSE 8080
diff --git a/examples/quickstarts/helidon-standalone-quickstart-mp/README.md b/examples/quickstarts/helidon-standalone-quickstart-mp/README.md
index f0d0ad8e4e..2dcb0bdba6 100644
--- a/examples/quickstarts/helidon-standalone-quickstart-mp/README.md
+++ b/examples/quickstarts/helidon-standalone-quickstart-mp/README.md
@@ -106,13 +106,13 @@ Start the application:
 Build the "native" Docker Image
 
 ```
-docker build -t helidon-quickstart-mp-native -f Dockerfile.native .
+docker build -t helidon-standalone-quickstart-mp-native -f Dockerfile.native .
 ```
 
 Start the application:
 
 ```
-docker run --rm -p 8080:8080 helidon-quickstart-mp-native:latest
+docker run --rm -p 8080:8080 helidon-standalone-quickstart-mp-native:latest
 ```
   
 ## Build a Java Runtime Image using jlink
@@ -141,7 +141,7 @@ See https://github.com/oracle/helidon-build-tools/tree/master/helidon-maven-plug
 Start the application:
 
 ```
-./target/helidon-quickstart-se-jri/bin/start
+./target/helidon-standalone-quickstart-mp-jri/bin/start
 ```
 
 ### Multi-stage Docker build
@@ -149,17 +149,17 @@ Start the application:
 Build the JRI as a Docker Image
 
 ```
-docker build -t helidon-quickstart-mp-jri -f Dockerfile.jlink .
+docker build -t helidon-standalone-quickstart-mp-jri -f Dockerfile.jlink .
 ```
 
 Start the application:
 
 ```
-docker run --rm -p 8080:8080 helidon-quickstart-mp-jri:latest
+docker run --rm -p 8080:8080 helidon-standalone-quickstart-mp-jri:latest
 ```
 
 See the start script help:
 
 ```
-docker run --rm helidon-quickstart-mp-jri:latest --help
+docker run --rm helidon-standalone-quickstart-mp-jri:latest --help
 ```
diff --git a/examples/quickstarts/helidon-standalone-quickstart-se/Dockerfile b/examples/quickstarts/helidon-standalone-quickstart-se/Dockerfile
index 14333cfde3..28188f2b5b 100644
--- a/examples/quickstarts/helidon-standalone-quickstart-se/Dockerfile
+++ b/examples/quickstarts/helidon-standalone-quickstart-se/Dockerfile
@@ -15,7 +15,7 @@
 #
 
 # 1st stage, build the app
-FROM container-registry.oracle.com/java/openjdk:21 as maven
+FROM container-registry.oracle.com/java/openjdk:21 as build
 
 # Install maven
 WORKDIR /usr/share
diff --git a/examples/quickstarts/helidon-standalone-quickstart-se/Dockerfile.jlink b/examples/quickstarts/helidon-standalone-quickstart-se/Dockerfile.jlink
index 7906e826c2..8d897eff91 100644
--- a/examples/quickstarts/helidon-standalone-quickstart-se/Dockerfile.jlink
+++ b/examples/quickstarts/helidon-standalone-quickstart-se/Dockerfile.jlink
@@ -15,7 +15,7 @@
 #
 
 # 1st stage, build the app
-FROM container-registry.oracle.com/java/openjdk:21 as maven
+FROM container-registry.oracle.com/java/openjdk:21 as build
 
 # Install maven
 WORKDIR /usr/share
@@ -44,6 +44,6 @@ RUN echo "done!"
 
 FROM debian:stretch-slim
 WORKDIR /helidon
-COPY --from=build /helidon/target/helidon-quickstart-se-jri ./
+COPY --from=build /helidon/target/helidon-standalone-quickstart-se-jri ./
 ENTRYPOINT ["/bin/bash", "/helidon/bin/start"]
 EXPOSE 8080
diff --git a/examples/quickstarts/helidon-standalone-quickstart-se/README.md b/examples/quickstarts/helidon-standalone-quickstart-se/README.md
index 73227cca7d..f7bd161fa4 100644
--- a/examples/quickstarts/helidon-standalone-quickstart-se/README.md
+++ b/examples/quickstarts/helidon-standalone-quickstart-se/README.md
@@ -142,7 +142,7 @@ See https://github.com/oracle/helidon-build-tools/tree/master/helidon-maven-plug
 Start the application:
 
 ```
-./target/helidon-quickstart-se-jri/bin/start
+./target/helidon-standalone-quickstart-se-jri/bin/start
 ```
 
 ### Multi-stage Docker build
@@ -150,17 +150,17 @@ Start the application:
 Build the JRI as a Docker Image
 
 ```
-docker build -t helidon-quickstart-se-jri -f Dockerfile.jlink .
+docker build -t helidon-standalone-quickstart-se-jri -f Dockerfile.jlink .
 ```
 
 Start the application:
 
 ```
-docker run --rm -p 8080:8080 helidon-quickstart-se-jri:latest
+docker run --rm -p 8080:8080 helidon-standalone-quickstart-se-jri:latest
 ```
 
 See the start script help:
 
 ```
-docker run --rm helidon-quickstart-se-jri:latest --help
+docker run --rm helidon-standalone-quickstart-se-jri:latest --help
 ```
diff --git a/examples/translator-app/README.md b/examples/translator-app/README.md
index 1a5f18afdb..e3913c42bc 100644
--- a/examples/translator-app/README.md
+++ b/examples/translator-app/README.md
@@ -28,7 +28,7 @@ kubectl apply \
 With Docker:
 ```bash
 docker build -t helidon-examples-translator-backend backend/
-docker build -t helidon-examples--translator-frontend frontend/
+docker build -t helidon-examples-translator-frontend frontend/
 docker run --rm -d -p 9080:9080 \
     --link zipkin \
     --name helidon-examples-translator-backend \
diff --git a/examples/translator-app/backend/Dockerfile b/examples/translator-app/backend/Dockerfile
index fd93dab321..41e657aed9 100644
--- a/examples/translator-app/backend/Dockerfile
+++ b/examples/translator-app/backend/Dockerfile
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2018, 2021 Oracle and/or its affiliates.
+# Copyright (c) 2018, 2023 Oracle and/or its affiliates.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,16 @@
 #
 
 # 1st stage, build the app
-FROM maven:3.6.3-openjdk-17-slim as build
+FROM container-registry.oracle.com/java/openjdk:21 as build
+
+# Install maven
+WORKDIR /usr/share
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
 
 WORKDIR /helidon
 
@@ -33,7 +42,7 @@ RUN mvn package -DskipTests
 RUN echo "done!"
 
 # 2nd stage, build the runtime image
-FROM openjdk:17-jdk-slim
+FROM container-registry.oracle.com/java/openjdk:21
 WORKDIR /helidon
 
 # Copy the binary built in the 1st stage
diff --git a/examples/translator-app/backend/pom.xml b/examples/translator-app/backend/pom.xml
index 53514032af..e7c937d9bc 100644
--- a/examples/translator-app/backend/pom.xml
+++ b/examples/translator-app/backend/pom.xml
@@ -29,7 +29,6 @@
     </parent>
     <groupId>io.helidon.examples.translator</groupId>
     <artifactId>helidon-examples-translator-backend</artifactId>
-    <version>4.0.0-SNAPSHOT</version>
     <name>Helidon Examples Translator Backend</name>
 
     <description>
diff --git a/examples/translator-app/frontend/Dockerfile b/examples/translator-app/frontend/Dockerfile
index c539e9f946..9d23350406 100644
--- a/examples/translator-app/frontend/Dockerfile
+++ b/examples/translator-app/frontend/Dockerfile
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2018, 2021 Oracle and/or its affiliates.
+# Copyright (c) 2018, 2023 Oracle and/or its affiliates.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,16 @@
 #
 
 # 1st stage, build the app
-FROM maven:3.6.3-openjdk-17-slim as build
+FROM container-registry.oracle.com/java/openjdk:21 as build
+
+# Install maven
+WORKDIR /usr/share
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
 
 WORKDIR /helidon
 
@@ -33,7 +42,7 @@ RUN mvn package -Dmaven.test.skip
 RUN echo "done!"
 
 # 2nd stage, build the runtime image
-FROM openjdk:17-jdk-slim
+FROM container-registry.oracle.com/java/openjdk:21
 WORKDIR /helidon
 
 # Copy the binary built in the 1st stage
diff --git a/examples/translator-app/frontend/pom.xml b/examples/translator-app/frontend/pom.xml
index c1c4c73957..8c632c1809 100644
--- a/examples/translator-app/frontend/pom.xml
+++ b/examples/translator-app/frontend/pom.xml
@@ -29,7 +29,6 @@
     </parent>
     <groupId>io.helidon.examples.translator</groupId>
     <artifactId>helidon-examples-translator-frontend</artifactId>
-    <version>4.0.0-SNAPSHOT</version>
     <name>Helidon Examples Translator Frontend</name>
 
     <description>
diff --git a/examples/webserver/opentracing/Dockerfile b/examples/webserver/opentracing/Dockerfile
index dccd442f15..47a15766dc 100644
--- a/examples/webserver/opentracing/Dockerfile
+++ b/examples/webserver/opentracing/Dockerfile
@@ -1,5 +1,5 @@
 #
-# Copyright (c) 2018, 2021 Oracle and/or its affiliates.
+# Copyright (c) 2018, 2023 Oracle and/or its affiliates.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -15,7 +15,16 @@
 #
 
 # 1st stage, build the app
-FROM maven:3.6.3-openjdk-17-slim as build
+FROM container-registry.oracle.com/java/openjdk:21 as build
+
+# Install maven
+WORKDIR /usr/share
+RUN set -x && \
+    curl -O https://archive.apache.org/dist/maven/maven-3/3.8.4/binaries/apache-maven-3.8.4-bin.tar.gz && \
+    tar -xvf apache-maven-*-bin.tar.gz  && \
+    rm apache-maven-*-bin.tar.gz && \
+    mv apache-maven-* maven && \
+    ln -s /usr/share/maven/bin/mvn /bin/
 
 WORKDIR /helidon
 
@@ -33,7 +42,7 @@ RUN mvn package -DskipTests
 RUN echo "done!"
 
 # 2nd stage, build the runtime image
-FROM openjdk:17-jdk-slim
+FROM container-registry.oracle.com/java/openjdk:21
 WORKDIR /helidon
 
 # Copy the binary built in the 1st stage
diff --git a/metrics/providers/micrometer/src/main/java/io/helidon/metrics/providers/micrometer/MicrometerPrometheusFormatter.java b/metrics/providers/micrometer/src/main/java/io/helidon/metrics/providers/micrometer/MicrometerPrometheusFormatter.java
index ebad78514a..c8aa27050f 100644
--- a/metrics/providers/micrometer/src/main/java/io/helidon/metrics/providers/micrometer/MicrometerPrometheusFormatter.java
+++ b/metrics/providers/micrometer/src/main/java/io/helidon/metrics/providers/micrometer/MicrometerPrometheusFormatter.java
@@ -131,10 +131,10 @@ public class MicrometerPrometheusFormatter implements MeterRegistryFormatter {
                 return Optional.empty();
             }
 
-            String prometheusOutput = filter(prometheusMeterRegistry.get()
-                                                     .scrape(MicrometerPrometheusFormatter.MEDIA_TYPE_TO_FORMAT.get(
-                                                                     resultMediaType),
-                                                             meterNamesOfInterest));
+            String prometheusOutput = prometheusMeterRegistry.get()
+                    .scrape(MicrometerPrometheusFormatter.MEDIA_TYPE_TO_FORMAT.get(
+                                    resultMediaType),
+                            meterNamesOfInterest);
 
             return prometheusOutput.isBlank() ? Optional.empty() : Optional.of(prometheusOutput);
         }
@@ -218,16 +218,6 @@ public class MicrometerPrometheusFormatter implements MeterRegistryFormatter {
         return result;
     }
 
-    /**
-     * Filter the Prometheus-format report.
-     *
-     * @param output Prometheus-format report
-     * @return output filtered
-     */
-    private static String filter(String output) {
-        return output.replaceFirst("# EOF\r?\n?", "");
-    }
-
     private static Optional<PrometheusMeterRegistry> prometheusMeterRegistry(MeterRegistry meterRegistry) {
         io.micrometer.core.instrument.MeterRegistry mMeterRegistry =
                 meterRegistry.unwrap(io.micrometer.core.instrument.MeterRegistry.class);
diff --git a/webserver/webserver/src/main/java/io/helidon/webserver/http1/Http1ServerResponse.java b/webserver/webserver/src/main/java/io/helidon/webserver/http1/Http1ServerResponse.java
index 5bf8ca2d18..81d39e4ca9 100644
--- a/webserver/webserver/src/main/java/io/helidon/webserver/http1/Http1ServerResponse.java
+++ b/webserver/webserver/src/main/java/io/helidon/webserver/http1/Http1ServerResponse.java
@@ -197,10 +197,14 @@ class Http1ServerResponse extends ServerResponseBase<Http1ServerResponse> {
 
         int writeBufferSize = ctx.listenerContext().config().writeBufferSize();
         outputStream = new ClosingBufferedOutputStream(bos, writeBufferSize);
+
+        OutputStream encodedOutputStream = contentEncode(outputStream);
+        // Headers can be augmented by encoders
+        bos.checkResponseHeaders();
         if (outputStreamFilter == null) {
-            return contentEncode(outputStream);
+            return encodedOutputStream;
         } else {
-            return outputStreamFilter.apply(contentEncode(outputStream));
+            return outputStreamFilter.apply(encodedOutputStream);
         }
     }
 
@@ -383,7 +387,7 @@ class Http1ServerResponse extends ServerResponseBase<Http1ServerResponse> {
         private final Http1ServerRequest request;
         private final boolean keepAlive;
         private final Supplier<String> streamResult;
-        private final boolean forcedChunked;
+        private boolean forcedChunked;
 
         private BufferData firstBuffer;
         private boolean closed;
@@ -414,13 +418,16 @@ class Http1ServerResponse extends ServerResponseBase<Http1ServerResponse> {
             this.responseCloseRunnable = responseCloseRunnable;
             this.ctx = ctx;
             this.sendListener = sendListener;
-            this.isChunked = !headers.contains(HeaderNames.CONTENT_LENGTH);
             this.contentLength = headers.contentLength().orElse(-1);
             this.request = request;
             this.keepAlive = keepAlive;
+            this.validateHeaders = validateHeaders;
+        }
+
+        void checkResponseHeaders(){
+            this.isChunked = !headers.contains(HeaderNames.CONTENT_LENGTH);
             this.forcedChunked = headers.contains(HeaderValues.TRANSFER_ENCODING_CHUNKED)
                     || headers.contains(HeaderNames.TRAILER);
-            this.validateHeaders = validateHeaders;
         }
 
         @Override


Test patch:
diff --git a/http/tests/encoding/gzip/pom.xml b/http/tests/encoding/gzip/pom.xml
index f9817ccba0..c7f8dc2da4 100644
--- a/http/tests/encoding/gzip/pom.xml
+++ b/http/tests/encoding/gzip/pom.xml
@@ -60,5 +60,10 @@
             <artifactId>hamcrest-all</artifactId>
             <scope>test</scope>
         </dependency>
+        <dependency>
+            <groupId>io.helidon.webserver</groupId>
+            <artifactId>helidon-webserver-static-content</artifactId>
+            <scope>test</scope>
+        </dependency>
     </dependencies>
 </project>
diff --git a/http/tests/encoding/gzip/src/test/java/io/helidon/http/tests/integration/encoding/gzip/GzipEncodingStaticContentTest.java b/http/tests/encoding/gzip/src/test/java/io/helidon/http/tests/integration/encoding/gzip/GzipEncodingStaticContentTest.java
new file mode 100644
index 0000000000..dc841af403
--- /dev/null
+++ b/http/tests/encoding/gzip/src/test/java/io/helidon/http/tests/integration/encoding/gzip/GzipEncodingStaticContentTest.java
@@ -0,0 +1,57 @@
+/*
+ * Copyright (c) 2023 Oracle and/or its affiliates.
+ *
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package io.helidon.http.tests.integration.encoding.gzip;
+
+import java.io.IOException;
+import java.net.URISyntaxException;
+import java.nio.file.Files;
+import java.nio.file.Path;
+
+import io.helidon.http.HeaderValues;
+import io.helidon.webclient.api.WebClient;
+import io.helidon.webserver.WebServerConfig;
+import io.helidon.webserver.http.HttpRouting;
+import io.helidon.webserver.staticcontent.StaticContentService;
+import io.helidon.webserver.testing.junit5.ServerTest;
+import io.helidon.webserver.testing.junit5.SetUpRoute;
+import io.helidon.webserver.testing.junit5.SetUpServer;
+
+import org.junit.jupiter.api.Test;
+
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.is;
+
+@ServerTest
+class GzipEncodingStaticContentTest {
+
+    @SetUpRoute
+    static void routing(HttpRouting.Builder builder) {
+        builder.register("/", StaticContentService.builder("/WEB")
+                .welcomeFileName("index.html")
+                .build());
+    }
+
+    @Test
+    void compressedStaticContent(WebClient client) throws IOException, URISyntaxException {
+        try (var res = client.get("/")
+                .header(HeaderValues.createCached("Accept-Encoding", "deflate, gzip"))
+                .request()) {
+            assertThat(res.as(String.class),
+                       is(Files.readString(Path.of(this.getClass().getResource("/WEB/index.html").toURI()))));
+        }
+    }
+}
diff --git a/http/tests/encoding/gzip/src/test/resources/WEB/index.html b/http/tests/encoding/gzip/src/test/resources/WEB/index.html
new file mode 100644
index 0000000000..826fb19718
--- /dev/null
+++ b/http/tests/encoding/gzip/src/test/resources/WEB/index.html
@@ -0,0 +1,27 @@
+<!--
+    Copyright (c) 2023 Oracle and/or its affiliates.
+
+    Licensed under the Apache License, Version 2.0 (the "License");
+    you may not use this file except in compliance with the License.
+    You may obtain a copy of the License at
+
+        http://www.apache.org/licenses/LICENSE-2.0
+
+    Unless required by applicable law or agreed to in writing, software
+    distributed under the License is distributed on an "AS IS" BASIS,
+    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+    See the License for the specific language governing permissions and
+    limitations under the License.
+
+  -->
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <title>Title</title>
+</head>
+<body>
+<h1>It works!</h1>
+
+</body>
+</html>
\ No newline at end of file
diff --git a/metrics/providers/micrometer/src/test/java/io/helidon/metrics/providers/micrometer/TestPrometheusFormatting.java b/metrics/providers/micrometer/src/test/java/io/helidon/metrics/providers/micrometer/TestPrometheusFormatting.java
index ac00f6e0c1..10a98c3103 100644
--- a/metrics/providers/micrometer/src/test/java/io/helidon/metrics/providers/micrometer/TestPrometheusFormatting.java
+++ b/metrics/providers/micrometer/src/test/java/io/helidon/metrics/providers/micrometer/TestPrometheusFormatting.java
@@ -19,6 +19,7 @@ import java.util.Optional;
 import java.util.Set;
 import java.util.concurrent.TimeUnit;
 
+import io.helidon.common.media.type.MediaTypes;
 import io.helidon.common.testing.junit5.OptionalMatcher;
 import io.helidon.metrics.api.Counter;
 import io.helidon.metrics.api.MeterRegistry;
@@ -36,6 +37,7 @@ import org.junit.jupiter.api.Test;
 import static org.hamcrest.MatcherAssert.assertThat;
 import static org.hamcrest.Matchers.allOf;
 import static org.hamcrest.Matchers.containsString;
+import static org.hamcrest.Matchers.endsWith;
 import static org.hamcrest.Matchers.instanceOf;
 import static org.hamcrest.Matchers.is;
 import static org.hamcrest.Matchers.not;
@@ -43,6 +45,13 @@ import static org.hamcrest.Matchers.not;
 class TestPrometheusFormatting {
 
     private static final String SCOPE_TAG_NAME = "this-scope";
+
+    /*
+    Only OpenMetrics format, not the Prometheus exposition format, has the trailing EOF which (for example) the Prometheus
+    server expects to see when it probes targets. That's why, below, when we format the output, we specify OpenMetrics as the
+    media type.
+     */
+    private static final String OPENMETRICS_EOF = "# EOF\n";
     private static MeterRegistry meterRegistry;
 
     private static MetricsConfig metricsConfig;
@@ -87,13 +96,15 @@ class TestPrometheusFormatting {
         e.record(2, TimeUnit.SECONDS);
 
         MicrometerPrometheusFormatter formatter = MicrometerPrometheusFormatter.builder(meterRegistry)
+                .resultMediaType(MediaTypes.APPLICATION_OPENMETRICS_TEXT)
                 .scopeTagName(SCOPE_TAG_NAME)
                 .build();
         Optional<Object> outputOpt = formatter.format();
 
         assertThat("Formatted output",
                    checkAndCast(outputOpt),
-                   allOf(containsString(scopeExpr("c1_total",
+                   allOf(
+                           containsString(scopeExpr("c1_total",
                                                   "this_scope",
                                                   "app",
                                                   "1.0")),
@@ -108,7 +119,9 @@ class TestPrometheusFormatting {
                          containsString(scopeExpr("t1_1_seconds_count",
                                                   "this_scope",
                                                   "app",
-                                                  "1.0"))));
+                                                  "1.0")),
+                         endsWith(OPENMETRICS_EOF)));
+
     }
 
     @Test
@@ -122,6 +135,7 @@ class TestPrometheusFormatting {
         d.record(7, TimeUnit.SECONDS);
 
         MicrometerPrometheusFormatter formatter = MicrometerPrometheusFormatter.builder(meterRegistry)
+                .resultMediaType(MediaTypes.APPLICATION_OPENMETRICS_TEXT)
                 .scopeTagName(SCOPE_TAG_NAME)
                 .meterNameSelection(Set.of("c2"))
                 .build();
@@ -139,7 +153,8 @@ class TestPrometheusFormatting {
                                                       "1.0"))),
                          not(containsString(scopeExpr("t2_seconds_sum",
                                                       "this_scope",
-                                                      "app", "7.0")))));
+                                                      "app", "7.0"))),
+                         endsWith(OPENMETRICS_EOF)));
 
     }
 
@@ -159,6 +174,7 @@ class TestPrometheusFormatting {
         e.record(2, TimeUnit.SECONDS);
 
         MicrometerPrometheusFormatter formatter = MicrometerPrometheusFormatter.builder(meterRegistry)
+                .resultMediaType(MediaTypes.APPLICATION_OPENMETRICS_TEXT)
                 .scopeTagName(SCOPE_TAG_NAME)
                 .scopeSelection(Set.of("app"))
                 .build();
@@ -182,11 +198,12 @@ class TestPrometheusFormatting {
                          containsString(scopeExpr("t3_1_seconds_count",
                                                   "this_scope",
                                                   "app",
-                                                  "1.0"))));
+                                                  "1.0")),
+                         endsWith(OPENMETRICS_EOF)));
     }
 
     private static String scopeExpr(String meterName, String key, String value, String suffix) {
-        return meterName + "{" + key + "=\"" + value + "\",} " + suffix;
+        return meterName + "{" + key + "=\"" + value + "\"} " + suffix;
     }
 
     private static String checkAndCast(Optional<Object> outputOpt) {