Repository: kubernetes/ingress-nginx
Pull request #8573 (2022-05-11 02:02:04)

Avoid race conditions by copying the list before sorting
When creating several ingresses at the same time a race condition can happen by modifying a variable deep in another object. When this race condition is triggered the generated nginx configuration is broken:

```
nginx: [emerg] invalid parameter "8.8.8.8/32,8" in /tmp/nginx-cfg4027854160:671
nginx: configuration file /tmp/nginx-cfg4027854160 test failed
```

Once it happens, the controller won't ever be able to generate the configuration again. Thus the only option is to restart the process.

There is not really a good way to reproduce this issue. It happens quite sporadically every 2 or 3 days. However, after this fix has been applied, we haven't seen it happen after about 4 weeks.

In our setup, in many of our services we run end-to-end tests as part of our CI/CD pipelines. When running the tests we deploy several services. At least two of them create together about 60 ingresses. Reaching over 2k ingresses in peak times.

The Ingress Controller has a list of default whitelist source range with about 35 IPs including a few IPv6. About 10% of the Ingresses created also contain their own set of whitelist source range varying from 4 to 12 IPs.

<img width="1608" alt="Screenshot 2022-04-28 at 13 20 39" src="https://user-images.githubusercontent.com/28046/167753834-2a326819-3bbf-474f-96e5-5e6676ecc4fa.png">

## Types of changes

- [x] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] Documentation only

## Checklist:

- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I've read the [CONTRIBUTION](https://github.com/kubernetes/ingress-nginx/blob/main/CONTRIBUTING.md) guide
- [ ] I have added tests to cover my changes.
- [x] All new and existing tests passed.


k8s-ci-robot: @renan: This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the `triage/accepted` label and provide further guidance.

The `triage/accepted` label can be added by org members by writing `/triage accepted` in a comment.


<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
k8s-ci-robot: Hi @renan. Thanks for your PR.

I'm waiting for a [kubernetes](https://github.com/orgs/kubernetes/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fingress-nginx).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/ingress-nginx/pull/8573#" title="Author self-approved">renan</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **strongjz** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @strongjz` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fingress-nginx).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes/ingress-nginx/blob/main/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["strongjz"]} -->
longwuyuan: Hi @renan ,

Thanks for your PR. I am not a developer but rough translation from  what changes are visible, it looks like your workaround is to create a backingArray for your whitelisted list of CIDR ranges and overwriting the state using that backingArray, later, to assure reconciled state.

If what I am saying above is roughly in the ballpark, then your problem description of creating 2K ingresses is less directly root-cause of the race condition, while a relatively big list  of whitelisted CIDR ranges, is more likely the root-cause of your race-condition. Do you have a very large list ?

This comment is to add enough information so that people wanting to work spend less time figuring out what is going on.

My request is this ;
- Create a issue
- Put the data in the issue as asked in the issue template
- Put the details of how many ipaddresses you have in whitelist
- Put the complete details of how you create 2K ingresses with complete information on precise commands and steps, in that issue
- Make sure not to miss the details of resources available to the cluster like cpu, memory etc
- Make sure not to miss details of your ingress-nginx controller use-case like deployment or daemonset or deployment with several replicas etc etc
- Put all other information there in that issue that may be related and that will help a developer here to work on this
- Link the issue to this PR
- Use the syntax `fixes #<issuenumber>`so that issue will close when/if this PR merges

If what I assumed above about whitelist CIDR ranges list is true, then the flip side of the story is that there is no check/limit on how many ipaddresses will be put in that list and created a new backingArray, and hence no clear idea on the impact. 

Most likely your changed code is not traversed, unless a user is using the whitelist annotation. So just curious to know what tests you have in mind, if any.

Once again, thank you for your contribution. There is lot of work being attempted on performance so this seems like a improvement for the mentioned use-case.

Patch:
diff --git a/.github/workflows/ci.yaml b/.github/workflows/ci.yaml
index e82d55134..cf8ec20f2 100644
--- a/.github/workflows/ci.yaml
+++ b/.github/workflows/ci.yaml
@@ -69,11 +69,11 @@ jobs:
       - name: Checkout
         uses: actions/checkout@629c2de402a417ea7690ca6ce3f33229e27606a5 # v2
 
-      - name: Set up Go 1.17
+      - name: Set up Go 1.18
         id: go
         uses: actions/setup-go@bfdd3570ce990073878bf10f6b2d79082de49492 # v2
         with:
-          go-version: '1.17.6'
+          go-version: '1.18.2'
 
       - name: Set up Docker Buildx
         id: buildx
@@ -133,7 +133,7 @@ jobs:
       - name: Setup Go
         uses: actions/setup-go@bfdd3570ce990073878bf10f6b2d79082de49492 # v2
         with:
-          go-version: '1.17.6'
+          go-version: '1.18.2'
 
       - name: cache
         uses: actions/download-artifact@f023be2c48cc18debc3bacd34cb396e0295e2869 # v2
@@ -403,12 +403,12 @@ jobs:
           version: v0.11.1
           image: kindest/node:v1.21.1
 
-      - name: Set up Go 1.17
+      - name: Set up Go 1.18
         id: go
         if: ${{ steps.filter-images.outputs.kube-webhook-certgen == 'true' }}
         uses: actions/setup-go@bfdd3570ce990073878bf10f6b2d79082de49492 # v2
         with:
-          go-version: '1.17.6'
+          go-version: '1.18.2'
 
       - name: kube-webhook-certgen image build
         if: ${{ steps.filter-images.outputs.kube-webhook-certgen == 'true' }}
diff --git a/RELEASE.md b/RELEASE.md
index 2d8e141aa..0dfdb8681 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -270,6 +270,7 @@ Promoting the images basically means that images, that were pushed to staging co
 
 - Update the version in [docs/deploy/index.md](docs/deploy/index.md)
 - Update Supported versions in the Support Versions table in the README.md
+- Execute the script to update e2e docs [hack/generate-e2e-suite-doc.sh](https://github.com/kubernetes/ingress-nginx/blob/main/hack/generate-e2e-suite-doc.sh)
 
 ### h. Edit stable.txt
 
diff --git a/charts/ingress-nginx/templates/controller-daemonset.yaml b/charts/ingress-nginx/templates/controller-daemonset.yaml
index 99882d1a8..2dca8e5c1 100644
--- a/charts/ingress-nginx/templates/controller-daemonset.yaml
+++ b/charts/ingress-nginx/templates/controller-daemonset.yaml
@@ -82,14 +82,7 @@ spec:
         {{- end }}
           args:
             {{- include "ingress-nginx.params" . | nindent 12 }}
-          securityContext:
-            capabilities:
-                drop:
-                - ALL
-                add:
-                - NET_BIND_SERVICE
-            runAsUser: {{ .Values.controller.image.runAsUser }}
-            allowPrivilegeEscalation: {{ .Values.controller.image.allowPrivilegeEscalation }}
+          securityContext: {{ include "controller.containerSecurityContext" . | nindent 12 }}
           env:
             - name: POD_NAME
               valueFrom:
diff --git a/charts/ingress-nginx/templates/controller-psp.yaml b/charts/ingress-nginx/templates/controller-psp.yaml
index a859594d1..fe34408c8 100644
--- a/charts/ingress-nginx/templates/controller-psp.yaml
+++ b/charts/ingress-nginx/templates/controller-psp.yaml
@@ -12,6 +12,9 @@ metadata:
 spec:
   allowedCapabilities:
     - NET_BIND_SERVICE
+  {{- if .Values.controller.image.chroot }}
+    - SYS_CHROOT
+  {{- end }}
 {{- if .Values.controller.sysctls }}
   allowedUnsafeSysctls:
   {{- range $sysctl, $value := .Values.controller.sysctls }}
diff --git a/cmd/nginx/flags.go b/cmd/nginx/flags.go
index 337a1f827..9a00b58b0 100644
--- a/cmd/nginx/flags.go
+++ b/cmd/nginx/flags.go
@@ -163,6 +163,9 @@ Requires the update-status parameter.`)
 			`Enables the collection of NGINX metrics`)
 		metricsPerHost = flags.Bool("metrics-per-host", true,
 			`Export metrics per-host`)
+		reportStatusClasses = flags.Bool("report-status-classes", false,
+			`Use status classes (2xx, 3xx, 4xx and 5xx) instead of status codes in metrics`)
+
 		timeBuckets         = flags.Float64Slice("time-buckets", prometheus.DefBuckets, "Set of buckets which will be used for prometheus histogram metrics such as RequestTime, ResponseTime")
 		lengthBuckets       = flags.Float64Slice("length-buckets", prometheus.LinearBuckets(10, 10, 10), "Set of buckets which will be used for prometheus histogram metrics such as RequestLength, ResponseLength")
 		sizeBuckets         = flags.Float64Slice("size-buckets", prometheus.ExponentialBuckets(10, 10, 7), "Set of buckets which will be used for prometheus histogram metrics such as BytesSent")
@@ -313,6 +316,7 @@ https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-g
 		EnableMetrics:               *enableMetrics,
 		MetricsPerHost:              *metricsPerHost,
 		MetricsBuckets:              histogramBuckets,
+		ReportStatusClasses:         *reportStatusClasses,
 		MonitorMaxBatchSize:         *monitorMaxBatchSize,
 		DisableServiceExternalName:  *disableServiceExternalName,
 		EnableSSLPassthrough:        *enableSSLPassthrough,
diff --git a/cmd/nginx/main.go b/cmd/nginx/main.go
index b8378d290..d5d6d91e7 100644
--- a/cmd/nginx/main.go
+++ b/cmd/nginx/main.go
@@ -133,7 +133,7 @@ func main() {
 
 	mc := metric.NewDummyCollector()
 	if conf.EnableMetrics {
-		mc, err = metric.NewCollector(conf.MetricsPerHost, reg, conf.IngressClassConfiguration.Controller, *conf.MetricsBuckets)
+		mc, err = metric.NewCollector(conf.MetricsPerHost, conf.ReportStatusClasses, reg, conf.IngressClassConfiguration.Controller, *conf.MetricsBuckets)
 		if err != nil {
 			klog.Fatalf("Error creating prometheus collector:  %v", err)
 		}
diff --git a/deploy/grafana/dashboards/nginx.json b/deploy/grafana/dashboards/nginx.json
index a2c277047..718d41a83 100644
--- a/deploy/grafana/dashboards/nginx.json
+++ b/deploy/grafana/dashboards/nginx.json
@@ -1016,6 +1016,7 @@
     {
       "columns": [],
       "datasource": "${DS_PROMETHEUS}",
+      "description": "This data is real time, independent of dashboard time range",
       "fontSize": "100%",
       "gridPos": {
         "h": 8,
@@ -1233,6 +1234,194 @@
       "transparent": false,
       "type": "table"
     },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "P1809F7CD0C75ACF3"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "color": {
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisLabel": "",
+            "axisPlacement": "auto",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "mappings": [],
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 80
+              }
+            ]
+          },
+          "unit": "s"
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 7,
+        "w": 12,
+        "x": 0,
+        "y": 24
+      },
+      "hideTimeOverride": false,
+      "id": 91,
+      "links": [],
+      "options": {
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom"
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "none"
+        }
+      },
+      "pluginVersion": "8.3.4",
+      "repeatDirection": "h",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P1809F7CD0C75ACF3"
+          },
+          "exemplar": true,
+          "expr": "histogram_quantile(0.80, sum(rate(nginx_ingress_controller_request_duration_seconds_bucket{ingress!=\"\",controller_pod=~\"$controller\",controller_class=~\"$controller_class\",controller_namespace=~\"$namespace\",ingress=~\"$ingress\",exported_namespace=\"uat\"}[2m])) by (le))",
+          "format": "time_series",
+          "hide": false,
+          "instant": false,
+          "interval": "",
+          "intervalFactor": 1,
+          "legendFormat": "P80",
+          "refId": "C"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P1809F7CD0C75ACF3"
+          },
+          "exemplar": true,
+          "expr": "histogram_quantile(0.90, sum(rate(nginx_ingress_controller_request_duration_seconds_bucket{ingress!=\"\",controller_pod=~\"$controller\",controller_class=~\"$controller_class\",controller_namespace=~\"$namespace\",ingress=~\"$ingress\",exported_namespace=\"uat\"}[2m])) by (le))",
+          "format": "time_series",
+          "hide": false,
+          "instant": false,
+          "interval": "",
+          "intervalFactor": 1,
+          "legendFormat": "P90",
+          "refId": "D"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P1809F7CD0C75ACF3"
+          },
+          "editorMode": "code",
+          "exemplar": true,
+          "expr": "histogram_quantile(0.99, sum(rate(nginx_ingress_controller_request_duration_seconds_bucket{ingress!=\"\",controller_pod=~\"$controller\",controller_class=~\"$controller_class\",controller_namespace=~\"$namespace\",ingress=~\"$ingress\",exported_namespace=\"uat\"}[2m])) by (le))",
+          "format": "time_series",
+          "hide": false,
+          "instant": false,
+          "interval": "",
+          "intervalFactor": 1,
+          "legendFormat": "P99",
+          "refId": "E"
+        }
+      ],
+      "title": "Ingress Percentile Response Times",
+      "type": "timeseries"
+    },
+    {
+      "cards": {},
+      "color": {
+        "cardColor": "#b4ff00",
+        "colorScale": "sqrt",
+        "colorScheme": "interpolateWarm",
+        "exponent": 0.5,
+        "mode": "spectrum"
+      },
+      "dataFormat": "tsbuckets",
+      "datasource": {
+        "type": "prometheus",
+        "uid": "P1809F7CD0C75ACF3"
+      },
+      "description": "",
+      "gridPos": {
+        "h": 7,
+        "w": 12,
+        "x": 12,
+        "y": 24
+      },
+      "heatmap": {},
+      "hideZeroBuckets": false,
+      "highlightCards": true,
+      "id": 89,
+      "legend": {
+        "show": true
+      },
+      "reverseYBuckets": false,
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "P1809F7CD0C75ACF3"
+          },
+          "exemplar": true,
+          "expr": "sum(increase(nginx_ingress_controller_request_duration_seconds_bucket{ingress!=\"\",controller_pod=~\"$controller\",controller_class=~\"$controller_class\",controller_namespace=~\"$namespace\",ingress=~\"$ingress\",exported_namespace=\"uat\"}[2m])) by (le)",
+          "format": "heatmap",
+          "interval": "",
+          "legendFormat": "{{le}}",
+          "refId": "A"
+        }
+      ],
+      "title": "Ingress Request Latency Heatmap",
+      "tooltip": {
+        "show": true,
+        "showHistogram": true
+      },
+      "type": "heatmap",
+      "xAxis": {
+        "show": true
+      },
+      "yAxis": {
+        "format": "s",
+        "logBase": 1,
+        "show": true
+      },
+      "yBucketBound": "auto"
+    },
     {
       "columns": [
         {
@@ -1246,7 +1435,7 @@
         "h": 8,
         "w": 24,
         "x": 0,
-        "y": 24
+        "y": 31
       },
       "height": "1024",
       "id": 85,
diff --git a/deploy/static/provider/kind/1.19/deploy.yaml b/deploy/static/provider/kind/1.19/deploy.yaml
index c61bcf9cd..ed44b79af 100644
--- a/deploy/static/provider/kind/1.19/deploy.yaml
+++ b/deploy/static/provider/kind/1.19/deploy.yaml
@@ -477,6 +477,9 @@ spec:
       - effect: NoSchedule
         key: node-role.kubernetes.io/master
         operator: Equal
+      - effect: NoSchedule
+        key: node-role.kubernetes.io/control-plane
+        operator: Equal
       volumes:
       - name: webhook-cert
         secret:
diff --git a/deploy/static/provider/kind/1.20/deploy.yaml b/deploy/static/provider/kind/1.20/deploy.yaml
index cda06a589..60fb24d4a 100644
--- a/deploy/static/provider/kind/1.20/deploy.yaml
+++ b/deploy/static/provider/kind/1.20/deploy.yaml
@@ -480,6 +480,9 @@ spec:
       - effect: NoSchedule
         key: node-role.kubernetes.io/master
         operator: Equal
+      - effect: NoSchedule
+        key: node-role.kubernetes.io/control-plane
+        operator: Equal
       volumes:
       - name: webhook-cert
         secret:
diff --git a/deploy/static/provider/kind/1.21/deploy.yaml b/deploy/static/provider/kind/1.21/deploy.yaml
index b3fac9bea..049796359 100644
--- a/deploy/static/provider/kind/1.21/deploy.yaml
+++ b/deploy/static/provider/kind/1.21/deploy.yaml
@@ -483,6 +483,9 @@ spec:
       - effect: NoSchedule
         key: node-role.kubernetes.io/master
         operator: Equal
+      - effect: NoSchedule
+        key: node-role.kubernetes.io/control-plane
+        operator: Equal
       volumes:
       - name: webhook-cert
         secret:
diff --git a/deploy/static/provider/kind/1.22/deploy.yaml b/deploy/static/provider/kind/1.22/deploy.yaml
index b3fac9bea..049796359 100644
--- a/deploy/static/provider/kind/1.22/deploy.yaml
+++ b/deploy/static/provider/kind/1.22/deploy.yaml
@@ -483,6 +483,9 @@ spec:
       - effect: NoSchedule
         key: node-role.kubernetes.io/master
         operator: Equal
+      - effect: NoSchedule
+        key: node-role.kubernetes.io/control-plane
+        operator: Equal
       volumes:
       - name: webhook-cert
         secret:
diff --git a/deploy/static/provider/kind/1.23/deploy.yaml b/deploy/static/provider/kind/1.23/deploy.yaml
index b3fac9bea..049796359 100644
--- a/deploy/static/provider/kind/1.23/deploy.yaml
+++ b/deploy/static/provider/kind/1.23/deploy.yaml
@@ -483,6 +483,9 @@ spec:
       - effect: NoSchedule
         key: node-role.kubernetes.io/master
         operator: Equal
+      - effect: NoSchedule
+        key: node-role.kubernetes.io/control-plane
+        operator: Equal
       volumes:
       - name: webhook-cert
         secret:
diff --git a/deploy/static/provider/kind/deploy.yaml b/deploy/static/provider/kind/deploy.yaml
index cda06a589..60fb24d4a 100644
--- a/deploy/static/provider/kind/deploy.yaml
+++ b/deploy/static/provider/kind/deploy.yaml
@@ -480,6 +480,9 @@ spec:
       - effect: NoSchedule
         key: node-role.kubernetes.io/master
         operator: Equal
+      - effect: NoSchedule
+        key: node-role.kubernetes.io/control-plane
+        operator: Equal
       volumes:
       - name: webhook-cert
         secret:
diff --git a/docs/deploy/index.md b/docs/deploy/index.md
index 283513ac7..0a33c19cd 100644
--- a/docs/deploy/index.md
+++ b/docs/deploy/index.md
@@ -28,7 +28,7 @@ On most Kubernetes clusters, the ingress controller will work without requiring
   - ... [Bare-metal](#bare-metal-clusters)
 - [Miscellaneous](#miscellaneous)
 
-## TODO : We have subdirectories for kubernetes versions now because of a PR https://github.com/kubernetes/ingress-nginx/pull/8162 . You can see this here https://github.com/kubernetes/ingress-nginx/tree/main/deploy/static/provider/cloud . We need to add documentation here that is clear and unambiguous in guiding users to pick the deployment manifest under a subdirectory, based on the K8S version being used. But until the explicit clear docs land here, users are recommended to feel free to use those subdirectories and get the manifest(s) related to their K8S version.
+<!-- TODO: We have subdirectories for kubernetes versions now because of a PR https://github.com/kubernetes/ingress-nginx/pull/8162 . You can see this here https://github.com/kubernetes/ingress-nginx/tree/main/deploy/static/provider/cloud . We need to add documentation here that is clear and unambiguous in guiding users to pick the deployment manifest under a subdirectory, based on the K8S version being used. But until the explicit clear docs land here, users are recommended to feel free to use those subdirectories and get the manifest(s) related to their K8S version. -->
 
 ## Quick start
 
diff --git a/docs/user-guide/cli-arguments.md b/docs/user-guide/cli-arguments.md
index f2b0f7a82..ae7c5b3ba 100644
--- a/docs/user-guide/cli-arguments.md
+++ b/docs/user-guide/cli-arguments.md
@@ -47,6 +47,7 @@ They are set in the container spec of the `ingress-nginx-controller` Deployment
 | `--publish-service`                | Service fronting the Ingress controller. Takes the form "namespace/name". When used together with update-status, the controller mirrors the address of this service's endpoints to the load-balancer status of all Ingress objects it satisfies. |
 | `--publish-status-address`         | Customized address (or addresses, separated by comma) to set as the load-balancer status of Ingress objects this controller satisfies. Requires the update-status parameter. |
 | `--report-node-internal-ip-address`| Set the load-balancer status of Ingress objects to internal Node addresses instead of external. Requires the update-status parameter. |
+| `--report-status-classes`          | If true, report status classes in metrics (2xx, 3xx, 4xx and 5xx) instead of full status codes. (default false) |
 | `--skip_headers`                   | If true, avoid header prefixes in the log messages |
 | `--skip_log_headers`               | If true, avoid headers when opening log files |
 | `--ssl-passthrough-proxy-port`     | Port to use internally for SSL Passthrough. (default 442) |
diff --git a/docs/user-guide/nginx-configuration/annotations.md b/docs/user-guide/nginx-configuration/annotations.md
index 2a019ea27..0c65bd0c8 100755
--- a/docs/user-guide/nginx-configuration/annotations.md
+++ b/docs/user-guide/nginx-configuration/annotations.md
@@ -488,6 +488,8 @@ Additionally it is possible to set:
   `<Cache_Key>` this enables caching for auth requests. specify a lookup key for auth responses. e.g. `$remote_user$http_authorization`. Each server and location has it's own keyspace. Hence a cached response is only valid on a per-server and per-location basis.
 * `nginx.ingress.kubernetes.io/auth-cache-duration`:
   `<Cache_duration>` to specify a caching time for auth responses based on their response codes, e.g. `200 202 30m`. See [proxy_cache_valid](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_valid) for details. You may specify multiple, comma-separated values: `200 202 10m, 401 5m`. defaults to `200 202 401 5m`.
+* `nginx.ingress.kubernetes.io/auth-always-set-cookie`:
+  `<Boolean_Flag>` to set a cookie returned by auth request. By default, the cookie will be set only if an upstream reports with the code 200, 201, 204, 206, 301, 302, 303, 304, 307, or 308.
 * `nginx.ingress.kubernetes.io/auth-snippet`:
   `<Auth_Snippet>` to specify a custom snippet to use with external authentication, e.g.
 
diff --git a/docs/user-guide/nginx-configuration/configmap.md b/docs/user-guide/nginx-configuration/configmap.md
index 57d79d536..6b6414218 100755
--- a/docs/user-guide/nginx-configuration/configmap.md
+++ b/docs/user-guide/nginx-configuration/configmap.md
@@ -1226,6 +1226,11 @@ Enables caching for global auth requests. Specify a lookup key for auth response
 
 Set a caching time for auth responses based on their response codes, e.g. `200 202 30m`. See [proxy_cache_valid](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_cache_valid) for details. You may specify multiple, comma-separated values: `200 202 10m, 401 5m`. defaults to `200 202 401 5m`.
 
+## global-auth-always-set-cookie
+
+Always set a cookie returned by auth request. By default, the cookie will be set only if an upstream reports with the code 200, 201, 204, 206, 301, 302, 303, 304, 307, or 308.
+_**default:**_ false
+
 ## no-auth-locations
 
 A comma-separated list of locations that should not get authenticated.
diff --git a/go.mod b/go.mod
index b13ed1cc6..8547d6970 100644
--- a/go.mod
+++ b/go.mod
@@ -1,6 +1,6 @@
 module k8s.io/ingress-nginx
 
-go 1.17
+go 1.18
 
 require (
 	github.com/armon/go-proxyproto v0.0.0-20210323213023-7e956b284f0a
@@ -16,9 +16,9 @@ require (
 	github.com/moul/pb v0.0.0-20180404114147-54bdd96e6a52
 	github.com/ncabatoff/process-exporter v0.7.10
 	github.com/onsi/ginkgo v1.16.5
-	github.com/opencontainers/runc v1.1.1
+	github.com/opencontainers/runc v1.1.2
 	github.com/pmezard/go-difflib v1.0.0
-	github.com/prometheus/client_golang v1.12.1
+	github.com/prometheus/client_golang v1.12.2
 	github.com/prometheus/client_model v0.2.0
 	github.com/prometheus/common v0.34.0
 	github.com/spf13/cobra v1.4.0
@@ -27,7 +27,7 @@ require (
 	github.com/zakjan/cert-chain-resolver v0.0.0-20211122211144-c6b0b792af9a
 	golang.org/x/crypto v0.0.0-20210817164053-32db794688a5
 	golang.org/x/net v0.0.0-20220225172249-27dd8689420f
-	google.golang.org/grpc v1.46.0
+	google.golang.org/grpc v1.46.2
 	gopkg.in/go-playground/pool.v3 v3.1.1
 	gopkg.in/mcuadros/go-syslog.v2 v2.3.0
 	k8s.io/api v0.23.6
diff --git a/go.sum b/go.sum
index c0e977ba5..6dca2bc83 100644
--- a/go.sum
+++ b/go.sum
@@ -96,7 +96,6 @@ github.com/blang/semver v3.5.1+incompatible/go.mod h1:kRBLl5iJ+tD4TcOOxsy/0fnweb
 github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
 github.com/certifi/gocertifi v0.0.0-20191021191039-0944d244cd40/go.mod h1:sGbDF6GwGcLpkNXPUTkMRoywsNa/ol15pxFe6ERfguA=
 github.com/certifi/gocertifi v0.0.0-20200922220541-2c3bb06c6054/go.mod h1:sGbDF6GwGcLpkNXPUTkMRoywsNa/ol15pxFe6ERfguA=
-github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
 github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
 github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE=
@@ -125,7 +124,6 @@ github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc
 github.com/coreos/go-oidc v2.1.0+incompatible/go.mod h1:CgnwVTmzoESiwO9qyAFEMiHoZ1nMCKZlZ9V6mm3/LKc=
 github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
 github.com/coreos/go-semver v0.3.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
-github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e h1:Wf6HqHfScWJN9/ZjdUKyjop4mf3Qdd+1TvvltAvM3m8=
 github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
 github.com/coreos/go-systemd/v22 v22.3.2 h1:D9/bQk5vlXQFZ6Kwuu6zaiXJ9oTPe68++AzAJc1DzSI=
 github.com/coreos/go-systemd/v22 v22.3.2/go.mod h1:Y58oyj3AT4RCenI/lSvhwexgC+NSVTIJ3seZv2GcEnc=
@@ -162,7 +160,6 @@ github.com/envoyproxy/go-control-plane v0.9.9-0.20210217033140-668b12f5399d/go.m
 github.com/envoyproxy/go-control-plane v0.9.9-0.20210512163311-63b5d3c536b0/go.mod h1:hliV/p42l8fGbc6Y9bQ70uLwIvmJyVE5k4iMKlh8wCQ=
 github.com/envoyproxy/go-control-plane v0.10.2-0.20220325020618-49ff273808a1/go.mod h1:KJwIaB5Mv44NWtYuAOFCVOjcI94vtpEz2JU/D2v6IjE=
 github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
-github.com/evanphx/json-patch v0.5.2/go.mod h1:ZWS5hhDbVDyob71nXKNL0+PWn6ToqBHMikGIFbs31qQ=
 github.com/evanphx/json-patch v4.11.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
 github.com/evanphx/json-patch v4.12.0+incompatible h1:4onqiflcdA9EOZ4RxV643DvftH5pOlLGNtQ5lPWQu84=
 github.com/evanphx/json-patch v4.12.0+incompatible/go.mod h1:50XU6AFN0ol/bzJsmQLiYLvXMP4fmwYFNcr97nuDLSk=
@@ -178,7 +175,6 @@ github.com/form3tech-oss/jwt-go v3.2.3+incompatible/go.mod h1:pbq4aXjuKjdthFRnoD
 github.com/frankban/quicktest v1.11.3/go.mod h1:wRf/ReqHper53s+kmmSZizM8NamnL3IM0I9ntUbOk+k=
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
-github.com/fsnotify/fsnotify v1.5.1/go.mod h1:T3375wBYaZdLLcVNkcVbzGHY7f1l/uK5T5Ai1i3InKU=
 github.com/fsnotify/fsnotify v1.5.4 h1:jRbGcIw6P2Meqdwuo0H1p6JVLbL5DHKAKlYndzMwVZI=
 github.com/fsnotify/fsnotify v1.5.4/go.mod h1:OVB6XrOHzAwXMpEM7uPOzcehqUV2UqJxmVXmkdnm1bU=
 github.com/fullsailor/pkcs7 v0.0.0-20160414161337-2585af45975b h1:074/xhloHUBOpTZwlIzQ28rbPY8pNJvzY7Gcx5KnNOk=
@@ -355,7 +351,6 @@ github.com/imkira/go-interpol v1.0.0 h1:HrmLyvOLJyjR0YofMw8QGdCIuYOs4TJUBDNU5sJC
 github.com/imkira/go-interpol v1.0.0/go.mod h1:z0h2/2T3XF8kyEPpRgJ3kmNv+C43p+I/CoI+jC3w2iA=
 github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
 github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
-github.com/jessevdk/go-flags v1.4.0/go.mod h1:4FA24M0QyGHXBuZZK/XkWh8h0e1EYbRYJSGM75WSRxI=
 github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
 github.com/jonboulle/clockwork v0.2.2/go.mod h1:Pkfl5aHPm1nk2H9h0bjmnJD/BcgbGXUBGnn1kMkgxc8=
 github.com/josharian/intern v1.0.0 h1:vlS4z54oSdjm0bgjRigI+G1HpF+tI+9rE5LLzOg8HmY=
@@ -466,7 +461,6 @@ github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+W
 github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
 github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
 github.com/onsi/ginkgo v1.14.0/go.mod h1:iSB4RoI2tjJc9BBv4NKIKWKya62Rps+oPG/Lv9klQyY=
-github.com/onsi/ginkgo v1.16.4/go.mod h1:dX+/inL/fNMqNlz0e9LfyB9TswhZpCVdJM/Z6Vvnwo0=
 github.com/onsi/ginkgo v1.16.5 h1:8xi0RTUf59SOSfEtZMvwTvXYMzG4gV23XVHOZiXNtnE=
 github.com/onsi/ginkgo v1.16.5/go.mod h1:+E8gABHa3K6zRBolWtd+ROzc/U5bkGt0FwiG042wbpU=
 github.com/onsi/gomega v0.0.0-20170829124025-dcabb60a477c/go.mod h1:C1qb7wdrVGGVU+Z6iS04AVkA3Q65CEZX59MT0QO5uiA=
@@ -474,9 +468,8 @@ github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1Cpa
 github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
 github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
 github.com/onsi/gomega v1.17.0 h1:9Luw4uT5HTjHTN8+aNcSThgH1vdXnmdJ8xIfZ4wyTRE=
-github.com/onsi/gomega v1.17.0/go.mod h1:HnhC7FXeEQY45zxNK3PPoIUhzk/80Xly9PcubAlGdZY=
-github.com/opencontainers/runc v1.1.1 h1:PJ9DSs2sVwE0iVr++pAHE6QkS9tzcVWozlPifdwMgrU=
-github.com/opencontainers/runc v1.1.1/go.mod h1:Tj1hFw6eFWp/o33uxGf5yF2BX5yz2Z6iptFpuvbbKqc=
+github.com/opencontainers/runc v1.1.2 h1:2VSZwLx5k/BfsBxMMipG/LYUnmqOD/BPkIVgQUcTlLw=
+github.com/opencontainers/runc v1.1.2/go.mod h1:Tj1hFw6eFWp/o33uxGf5yF2BX5yz2Z6iptFpuvbbKqc=
 github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417 h1:3snG66yBm59tKhhSPQrQ/0bCrv1LQbKt40LnUPiUxdc=
 github.com/opencontainers/runtime-spec v1.0.3-0.20210326190908-1c3f411f0417/go.mod h1:jwyrGlmzljRJv/Fgzds9SsS/C5hL+LL3ko9hs6T5lQ0=
 github.com/opencontainers/selinux v1.10.0/go.mod h1:2i0OySw99QjzBBQByd1Gr9gSjvuho1lHsJxIJ3gGbJI=
@@ -501,8 +494,9 @@ github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDf
 github.com/prometheus/client_golang v1.0.0/go.mod h1:db9x61etRT2tGnBNRi70OPL5FsnadC4Ky3P0J6CfImo=
 github.com/prometheus/client_golang v1.7.1/go.mod h1:PY5Wy2awLA44sXw4AOSfFBetzPP4j5+D6mVACh+pe2M=
 github.com/prometheus/client_golang v1.11.0/go.mod h1:Z6t4BnS23TR94PD6BsDNk8yVqroYurpAkEiz0P2BEV0=
-github.com/prometheus/client_golang v1.12.1 h1:ZiaPsmm9uiBeaSMRznKsCDNtPCS0T3JVDGF+06gjBzk=
 github.com/prometheus/client_golang v1.12.1/go.mod h1:3Z9XVyYiZYEO+YQWt3RD2R3jrbd179Rt297l4aS6nDY=
+github.com/prometheus/client_golang v1.12.2 h1:51L9cDoUHVrXx4zWYlcLQIZ+d+VXHgqnYKkIuq4g/34=
+github.com/prometheus/client_golang v1.12.2/go.mod h1:3Z9XVyYiZYEO+YQWt3RD2R3jrbd179Rt297l4aS6nDY=
 github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
 github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
 github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
@@ -662,8 +656,6 @@ go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
 go.uber.org/atomic v1.7.0 h1:ADUqmZGgLDDfbSL9ZmPxKTybcoEYHgpYfELNoN+7hsw=
 go.uber.org/atomic v1.7.0/go.mod h1:fEN4uk6kAWBTFdckzkM89CLk9XfWZrxpCo0nPH17wJc=
 go.uber.org/goleak v1.1.10/go.mod h1:8a7PlsEVH3e/a/GLqe5IIrQx6GzcnRmZEufDUTk4A7A=
-go.uber.org/goleak v1.1.11-0.20210813005559-691160354723/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
-go.uber.org/goleak v1.1.12/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
 go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
 go.uber.org/multierr v1.6.0 h1:y6IPFStTAIT5Ytl7/XYmHvzXQ7S3g/IeZW9hyZ5thw4=
 go.uber.org/multierr v1.6.0/go.mod h1:cdWPpRnG4AhwMwsgIHip0KRBQjJy5kYEpYjJxpXp9iU=
@@ -671,7 +663,6 @@ go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
 go.uber.org/zap v1.17.0/go.mod h1:MXVU+bhUf/A7Xi2HNOnopQOrmycQ5Ih87HtOu4q5SSo=
 go.uber.org/zap v1.19.0/go.mod h1:xg/QME4nWcxGxrpdeYfq7UvYrLh66cuVKdrbD1XF/NI=
 go.uber.org/zap v1.19.1 h1:ue41HOKd1vGURxrmeKIgELGb3jPW9DMUDGtsinblHwI=
-go.uber.org/zap v1.19.1/go.mod h1:j3DNczoxDZroyBnOT1L/Q79cfUMGZxlv/9dzN7SM1rI=
 golang.org/dl v0.0.0-20190829154251-82a15e2f2ead/go.mod h1:IUMfjQLJQd4UTqG1Z90tenwKoCX93Gn3MAQJMOSBsDQ=
 golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
 golang.org/x/crypto v0.0.0-20181029021203-45a5f77698d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
@@ -767,7 +758,6 @@ golang.org/x/net v0.0.0-20210119194325-5f4716e94777/go.mod h1:m0MpNAwzfU5UDzcl9v
 golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
 golang.org/x/net v0.0.0-20210316092652-d523dce5a7f4/go.mod h1:RBQZq4jEuRlivfhVLdyRGr576XBO4/greRjx4P4O3yc=
 golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
-golang.org/x/net v0.0.0-20210428140749-89ef3d95e781/go.mod h1:OJAsFXCWl8Ukc7SiCT/9KSuxbyM7479/AVlXFRxuMCk=
 golang.org/x/net v0.0.0-20210510120150-4163338589ed/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20210525063256-abc453219eb5/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
 golang.org/x/net v0.0.0-20210805182204-aaa1db679c0d/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
@@ -869,12 +859,10 @@ golang.org/x/sys v0.0.0-20210514084401-e8d321eab015/go.mod h1:oPkhp1MJrh7nUepCBc
 golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210616094352-59db8d763f22/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210809222454-d867a43fc93e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210831042530-f4d43177bf5e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20210906170528-6f6e22806c34/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20211025201205-69cdffdb9359/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
-golang.org/x/sys v0.0.0-20211029165221-6e7872819dc8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20211116061358-0a5406a5449c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/sys v0.0.0-20220114195835-da31bd327af9/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
@@ -966,7 +954,6 @@ golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8T
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 gomodules.xyz/jsonpatch/v2 v2.2.0 h1:4pT439QV83L+G9FkcCriY6EkpcK6r6bK+A5FBUMI7qY=
-gomodules.xyz/jsonpatch/v2 v2.2.0/go.mod h1:WXp+iVDkoLQqPudfQ9GBlwB2eZ5DKOnjQZCYdOS8GPY=
 google.golang.org/api v0.4.0/go.mod h1:8k5glujaEP+g9n7WNsDg8QP6cUVNI86fCNMcbazEtwE=
 google.golang.org/api v0.7.0/go.mod h1:WtwebWUNSVBH/HAw79HIFXZNqEvBhG+Ra+ax0hx3E3M=
 google.golang.org/api v0.8.0/go.mod h1:o4eAsZoiT+ibD93RtjEohWalFOjRDx6CVaqeizhEnKg=
@@ -1066,8 +1053,8 @@ google.golang.org/grpc v1.36.1/go.mod h1:qjiiYl8FncCW8feJPdyg3v6XW24KsRHe+dy9BAG
 google.golang.org/grpc v1.37.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM=
 google.golang.org/grpc v1.38.0/go.mod h1:NREThFqKR1f3iQ6oBuvc5LadQuXVGo9rkm5ZGrQdJfM=
 google.golang.org/grpc v1.40.0/go.mod h1:ogyxbiOoUXAkP+4+xa6PZSE9DZgIHtSpzjDTB9KAK34=
-google.golang.org/grpc v1.46.0 h1:oCjezcn6g6A75TGoKYBPgKmVBLexhYLM6MebdrPApP8=
-google.golang.org/grpc v1.46.0/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk=
+google.golang.org/grpc v1.46.2 h1:u+MLGgVf7vRdjEYZ8wDFhAVNmhkbJ5hmrA1LMWK1CAQ=
+google.golang.org/grpc v1.46.2/go.mod h1:vN9eftEi1UMyUsIF80+uQXhHjbXYbm0uXoFCACuMGWk=
 google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
 google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
 google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
diff --git a/hack/manifest-templates/provider/kind/values.yaml b/hack/manifest-templates/provider/kind/values.yaml
index f327c351e..ed636f372 100644
--- a/hack/manifest-templates/provider/kind/values.yaml
+++ b/hack/manifest-templates/provider/kind/values.yaml
@@ -17,6 +17,9 @@ controller:
     - key: "node-role.kubernetes.io/master"
       operator: "Equal"
       effect: "NoSchedule"
+    - key: "node-role.kubernetes.io/control-plane"
+      operator: "Equal"
+      effect: "NoSchedule"
 
   publishService:
     enabled: false
diff --git a/images/custom-error-pages/rootfs/Dockerfile b/images/custom-error-pages/rootfs/Dockerfile
index cdceb032b..d53228201 100755
--- a/images/custom-error-pages/rootfs/Dockerfile
+++ b/images/custom-error-pages/rootfs/Dockerfile
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM golang:1.17-alpine as builder
+FROM golang:1.18.2-alpine as builder
 RUN apk add git
 
 WORKDIR /go/src/k8s.io/ingress-nginx/images/custom-error-pages
diff --git a/images/custom-error-pages/rootfs/go.mod b/images/custom-error-pages/rootfs/go.mod
index dec3b7bbb..05bd45400 100644
--- a/images/custom-error-pages/rootfs/go.mod
+++ b/images/custom-error-pages/rootfs/go.mod
@@ -1,6 +1,6 @@
 module k8s.io/ingress-nginx/custom-error-pages
 
-go 1.17
+go 1.18
 
 require github.com/prometheus/client_golang v1.11.0
 
diff --git a/images/ext-auth-example-authsvc/rootfs/Dockerfile b/images/ext-auth-example-authsvc/rootfs/Dockerfile
index ee2737286..9a52919be 100644
--- a/images/ext-auth-example-authsvc/rootfs/Dockerfile
+++ b/images/ext-auth-example-authsvc/rootfs/Dockerfile
@@ -1,4 +1,4 @@
-FROM golang:1.17.6-alpine3.15 as builder
+FROM golang:1.18.2-alpine3.15 as builder
 RUN mkdir /authsvc
 WORKDIR /authsvc
 COPY . ./
@@ -7,4 +7,4 @@ RUN CGO_ENABLED=0 GOOS=linux go build -o authsvc authsvc.go
 FROM gcr.io/distroless/base-debian11
 COPY --from=builder /authsvc/authsvc /
 EXPOSE 8080
-ENTRYPOINT ["/authsvc"]
\ No newline at end of file
+ENTRYPOINT ["/authsvc"]
diff --git a/images/ext-auth-example-authsvc/rootfs/go.mod b/images/ext-auth-example-authsvc/rootfs/go.mod
index 72f2a1049..052d17d6f 100644
--- a/images/ext-auth-example-authsvc/rootfs/go.mod
+++ b/images/ext-auth-example-authsvc/rootfs/go.mod
@@ -1,8 +1,7 @@
 module example.com/authsvc
 
-go 1.17
+go 1.18
 
-require (
-	github.com/google/uuid v1.1.2 // indirect
-	k8s.io/apimachinery v0.23.1 // indirect
-)
+require k8s.io/apimachinery v0.23.1
+
+require github.com/google/uuid v1.1.2 // indirect
diff --git a/images/kube-webhook-certgen/rootfs/Dockerfile b/images/kube-webhook-certgen/rootfs/Dockerfile
index f9020fa41..4a60ed2ea 100644
--- a/images/kube-webhook-certgen/rootfs/Dockerfile
+++ b/images/kube-webhook-certgen/rootfs/Dockerfile
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM --platform=$BUILDPLATFORM golang:1.16 as builder
+FROM --platform=$BUILDPLATFORM golang:1.18.2 as builder
 ARG BUILDPLATFORM
 ARG TARGETARCH
 
diff --git a/images/kube-webhook-certgen/rootfs/go.mod b/images/kube-webhook-certgen/rootfs/go.mod
index edd74bea3..3858ae566 100644
--- a/images/kube-webhook-certgen/rootfs/go.mod
+++ b/images/kube-webhook-certgen/rootfs/go.mod
@@ -1,14 +1,48 @@
 module github.com/jet/kube-webhook-certgen
 
-go 1.16
+go 1.18
 
 require (
 	github.com/onrik/logrus v0.9.0
 	github.com/sirupsen/logrus v1.8.1
 	github.com/spf13/cobra v1.1.3
-	github.com/tidwall/gjson v1.14.0 // indirect
 	k8s.io/api v0.22.6
 	k8s.io/apimachinery v0.22.6
 	k8s.io/client-go v0.22.6
 	k8s.io/kube-aggregator v0.22.6
 )
+
+require (
+	github.com/davecgh/go-spew v1.1.1 // indirect
+	github.com/evanphx/json-patch v4.11.0+incompatible // indirect
+	github.com/go-logr/logr v0.4.0 // indirect
+	github.com/gogo/protobuf v1.3.2 // indirect
+	github.com/golang/protobuf v1.5.2 // indirect
+	github.com/google/go-cmp v0.5.5 // indirect
+	github.com/google/gofuzz v1.1.0 // indirect
+	github.com/googleapis/gnostic v0.5.5 // indirect
+	github.com/imdario/mergo v0.3.5 // indirect
+	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/json-iterator/go v1.1.11 // indirect
+	github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
+	github.com/modern-go/reflect2 v1.0.1 // indirect
+	github.com/pkg/errors v0.9.1 // indirect
+	github.com/spf13/pflag v1.0.5 // indirect
+	github.com/tidwall/gjson v1.14.0 // indirect
+	golang.org/x/net v0.0.0-20211209124913-491a49abca63 // indirect
+	golang.org/x/oauth2 v0.0.0-20200107190931-bf48bf16ab8d // indirect
+	golang.org/x/sys v0.0.0-20210616094352-59db8d763f22 // indirect
+	golang.org/x/term v0.0.0-20210220032956-6a3ed077a48d // indirect
+	golang.org/x/text v0.3.6 // indirect
+	golang.org/x/time v0.0.0-20210723032227-1f47c861a9ac // indirect
+	google.golang.org/appengine v1.6.5 // indirect
+	google.golang.org/protobuf v1.26.0 // indirect
+	gopkg.in/inf.v0 v0.9.1 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
+	gopkg.in/yaml.v3 v3.0.0-20210107192922-496545a6307b // indirect
+	k8s.io/klog/v2 v2.9.0 // indirect
+	k8s.io/kube-openapi v0.0.0-20211109043538-20434351676c // indirect
+	k8s.io/utils v0.0.0-20210819203725-bdf08cb9a70a // indirect
+	sigs.k8s.io/structured-merge-diff/v4 v4.2.1 // indirect
+	sigs.k8s.io/yaml v1.2.0 // indirect
+)
diff --git a/images/opentelemetry/rootfs/CMakeLists.txt b/images/opentelemetry/rootfs/CMakeLists.txt
new file mode 100644
index 000000000..7278016a0
--- /dev/null
+++ b/images/opentelemetry/rootfs/CMakeLists.txt
@@ -0,0 +1,95 @@
+#!/bin/bash
+
+# Copyright 2021 The Kubernetes Authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+cmake_minimum_required(VERSION 3.11 FATAL_ERROR)
+
+project(
+  dependencies
+  LANGUAGES CXX
+  VERSION 0.0.1)
+
+set(CMAKE_CXX_STANDARD 11)
+set(CMAKE_CXX_EXTENSIONS OFF)
+set(CMAKE_CXX_STANDARD_REQUIRED ON)
+set(CMAKE_CXX_FLAGS "-O2")
+
+set(CMAKE_BUILD_TYPE
+    Release
+    CACHE STRING "Build type" FORCE)
+
+include(GNUInstallDirs)
+
+set(CMAKE_ARCHIVE_OUTPUT_DIRECTORY
+    ${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR})
+set(CMAKE_LIBRARY_OUTPUT_DIRECTORY
+    ${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_LIBDIR})
+set(CMAKE_RUNTIME_OUTPUT_DIRECTORY
+    ${PROJECT_BINARY_DIR}/${CMAKE_INSTALL_BINDIR})
+
+set(INSTALL_LIBDIR
+    ${CMAKE_INSTALL_LIBDIR}
+    CACHE PATH "directory for libraries")
+set(INSTALL_BINDIR
+    ${CMAKE_INSTALL_BINDIR}
+    CACHE PATH "directory for executables")
+set(INSTALL_INCLUDEDIR
+    ${CMAKE_INSTALL_INCLUDEDIR}
+    CACHE PATH "directory for header files")
+
+set(DEF_INSTALL_CMAKEDIR share/cmake/${PROJECT_NAME})
+set(INSTALL_CMAKEDIR
+    ${DEF_INSTALL_CMAKEDIR}
+    CACHE PATH "directory for CMake files")
+
+set_property(DIRECTORY PROPERTY EP_BASE ${CMAKE_BINARY_DIR}/subs)
+
+set(STAGED_INSTALL_PREFIX ${CMAKE_BINARY_DIR}/stage)
+message(STATUS "${PROJECT_NAME} staged install: ${STAGED_INSTALL_PREFIX}")
+
+find_package(OpenSSL REQUIRED)
+message("OpenSSL include dir: ${OPENSSL_INCLUDE_DIR}")
+message("OpenSSL libraries: ${OPENSSL_LIBRARIES}")
+
+set(GRPC_GIT_TAG
+    "v1.45.2"
+    CACHE STRING "gRPC version")
+
+include(ExternalProject)
+ExternalProject_Add(
+  gRPC
+  GIT_REPOSITORY https://github.com/grpc/grpc.git
+  GIT_TAG ${GRPC_GIT_TAG}
+  GIT_SHALLOW 1
+  UPDATE_COMMAND ""
+  CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${STAGED_INSTALL_PREFIX}
+             -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
+             -DCMAKE_CXX_COMPILER=${CMAKE_CXX_COMPILER}
+             -DgRPC_SSL_PROVIDER=package
+             -DOPENSSL_ROOT_DIR=OpenSSL
+             -DgRPC_BUILD_TESTS=OFF
+             -DBUILD_SHARED_LIBS=ON
+             -DgRPC_INSTALL=ON
+  CMAKE_CACHE_ARGS -DCMAKE_CXX_FLAGS:STRING=${CMAKE_CXX_FLAGS}
+  TEST_AFTER_INSTALL 0
+  DOWNLOAD_NO_PROGRESS 1
+  LOG_CONFIGURE 1
+  LOG_BUILD 0
+  LOG_INSTALL 1)
+
+install(
+  DIRECTORY ${STAGED_INSTALL_PREFIX}/
+  DESTINATION .
+  USE_SOURCE_PERMISSIONS)
diff --git a/images/opentelemetry/rootfs/Dockerfile b/images/opentelemetry/rootfs/Dockerfile
index ebf288c56..91832fa55 100644
--- a/images/opentelemetry/rootfs/Dockerfile
+++ b/images/opentelemetry/rootfs/Dockerfile
@@ -13,16 +13,33 @@
 # limitations under the License.
 
 
-FROM alpine:3.14.6 as builder
+FROM alpine:3.14.6 as base
 
-COPY . /
+RUN mkdir -p /opt/third_party/install
+COPY . /opt/third_party/
 
+# install build tools
 RUN apk update \
 	&& apk upgrade \
 	&& apk add -U bash \
-	&& /build.sh
+	&& bash /opt/third_party/build.sh -p
 
-FROM alpine:3.14.6 
+# install gRPC
+FROM base as grpc
+RUN bash /opt/third_party/build.sh -g v1.43.2
 
-COPY --from=builder init_module.sh /usr/local/bin/init_module.sh
-COPY --from=builder /etc/nginx/modules /etc/nginx/modules
+# install OpenTelemetry-cpp
+FROM base as otel-cpp
+COPY --from=grpc /opt/third_party/install/ /usr
+RUN bash /opt/third_party/build.sh -o v1.3.0
+
+# install otel_ngx_module.so
+FROM base as nginx
+COPY --from=grpc /opt/third_party/install/ /usr
+COPY --from=otel-cpp /opt/third_party/install/ /usr
+RUN bash /opt/third_party/build.sh -n
+
+FROM alpine:3.14.6
+COPY --from=base /opt/third_party/init_module.sh /usr/local/bin/init_module.sh
+COPY --from=nginx /etc/nginx/modules /etc/nginx/modules
+COPY --from=nginx /opt/third_party/install/lib /etc/nginx/modules
diff --git a/images/opentelemetry/rootfs/build.sh b/images/opentelemetry/rootfs/build.sh
index 67d1e67b2..91298274d 100755
--- a/images/opentelemetry/rootfs/build.sh
+++ b/images/opentelemetry/rootfs/build.sh
@@ -18,15 +18,12 @@ set -o errexit
 set -o nounset
 set -o pipefail
 
-export NGINX_VERSION=1.19.10
-
+export GRPC_GIT_TAG=${GRPC_GIT_TAG:="v1.43.2"}
 # Check for recent changes: https://github.com/open-telemetry/opentelemetry-cpp/compare/v1.2.0...main
-export OPENTELEMETRY_CPP_VERSION=1.2.0
-
-# Check for recent changes: https://github.com/open-telemetry/opentelemetry-cpp-contrib/compare/2656a4...main
-export OPENTELEMETRY_CONTRIB_COMMIT=2656a4072e257b6794da86ddd1b773b49f5517b3
-
-export BUILD_PATH=/tmp/build
+export OPENTELEMETRY_CPP_VERSION=${OPENTELEMETRY_CPP_VERSION:="1.2.0"}
+export INSTAL_DIR=/opt/third_party/install
+# improve compilation times
+CORES=$(($(grep -c ^processor /proc/cpuinfo) - 1))
 
 rm -rf \
    /var/cache/debconf/* \
@@ -35,15 +32,80 @@ rm -rf \
    /tmp/* \
    /var/tmp/*
 
-
-mkdir -p /etc/nginx
+export BUILD_PATH=/tmp/build
 mkdir --verbose -p "$BUILD_PATH"
-cd "$BUILD_PATH"
 
-apk add \
-  curl \
-  git \
-  build-base
+Help()
+{
+   # Display Help
+   echo "Add description of the script functions here."
+   echo
+   echo "Syntax: scriptTemplate [-h|g|o|n|p|]"
+   echo "options:"
+   echo "h     Print Help."
+   echo "g     gRPC git tag"
+   echo "o     OpenTelemetry git tag"
+   echo "n     install nginx"
+   echo "p     prepare"
+   echo
+}
+
+prepare()
+{
+  apk add \
+    linux-headers \
+    openssl \
+    curl-dev \
+    openssl-dev \
+    gtest-dev \
+    c-ares-dev \
+    pcre-dev \
+    curl \
+    git \
+    build-base
+}
+
+install_grpc()
+{
+  mkdir -p $BUILD_PATH/grpc
+  cd ${BUILD_PATH}/grpc
+  cmake -DCMAKE_INSTALL_PREFIX=${INSTAL_DIR} \
+    -DGRPC_GIT_TAG=${GRPC_GIT_TAG} /opt/third_party \
+    -DgRPC_BUILD_GRPC_NODE_PLUGIN=OFF \
+    -DgRPC_BUILD_GRPC_OBJECTIVE_C_PLUGIN=OFF \
+    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
+    -DgRPC_BUILD_GRPC_PHP_PLUGIN=OFF \
+    -DgRPC_BUILD_GRPC_PYTHON_PLUGIN=OFF \
+    -DgRPC_BUILD_GRPC_RUBY_PLUGIN=OFF
+  cmake --build . -j ${CORES} --target all install
+}
+
+install_otel()
+{
+  cd ${BUILD_PATH}
+  export LD_LIBRARY_PATH="${LD_LIBRARY_PATH:+LD_LIBRARY_PATH:}${INSTAL_DIR}/lib:/usr/local"
+  export PATH="${PATH}:${INSTAL_DIR}/bin"
+  git clone --recurse-submodules -j ${CORES} --depth=1 -b \
+    ${OPENTELEMETRY_CPP_VERSION} https://github.com/open-telemetry/opentelemetry-cpp.git opentelemetry-cpp-${OPENTELEMETRY_CPP_VERSION}
+  cd "opentelemetry-cpp-${OPENTELEMETRY_CPP_VERSION}"
+  mkdir -p .build
+  cd .build
+
+  cmake -DCMAKE_BUILD_TYPE=Release \
+        -DCMAKE_POSITION_INDEPENDENT_CODE=TRUE  \
+        -DWITH_ZIPKIN=OFF \
+        -DWITH_JAEGER=OFF \
+        -DCMAKE_INSTALL_PREFIX=${INSTAL_DIR} \
+        -DBUILD_TESTING=OFF \
+        -DBUILD_SHARED_LIBS=ON \
+        -DWITH_OTLP=ON \
+        -DWITH_OTLP_GRPC=ON \
+        -DWITH_EXAMPLES=OFF \
+        -DWITH_ABSEIL=ON \
+        -DWITH_OTLP_HTTP=OFF \
+        ..
+  cmake --build . -j ${CORES} --target install
+}
 
 get_src()
 {
@@ -53,59 +115,111 @@ get_src()
 
   echo "Downloading $url"
 
-  curl -sSL "$url" -o "$f"
+  curl -sSL --fail-with-body "$url" -o "$f"
   echo "$hash  $f" | sha256sum -c - || exit 10
   tar xzf "$f"
   rm -rf "$f"
 }
 
+install_nginx()
+{
+  export NGINX_VERSION=1.19.10
+
+  # Check for recent changes: https://github.com/open-telemetry/opentelemetry-cpp-contrib/compare/2656a4...main
+  export OPENTELEMETRY_CONTRIB_COMMIT=6467ec2e4d67b08b44580b7eb7a298786f4eef91
+
+  mkdir -p /etc/nginx
+  cd "$BUILD_PATH"
+
+  # TODO fix curl
+  # get_src 0528e793a97f942868616449d49326160f9cb67b2253fb2c4864603ac6ab09a9 \
+  #         "https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/$OPENTELEMETRY_CONTRIB_COMMIT.tar.gz"
+
+  git clone https://github.com/open-telemetry/opentelemetry-cpp-contrib.git \
+    opentelemetry-cpp-contrib-${OPENTELEMETRY_CONTRIB_COMMIT}
+  cd ${BUILD_PATH}/opentelemetry-cpp-contrib-${OPENTELEMETRY_CONTRIB_COMMIT}
+  git reset --hard ${OPENTELEMETRY_CONTRIB_COMMIT}
+  cd ${BUILD_PATH}/opentelemetry-cpp-contrib-${OPENTELEMETRY_CONTRIB_COMMIT}/instrumentation/nginx
+  mkdir -p build
+  cd build
+  cmake -DCMAKE_BUILD_TYPE=Release \
+    -DCMAKE_INSTALL_PREFIX=${INSTAL_DIR} \
+    -DBUILD_SHARED_LIBS=ON \
+    -DNGINX_VERSION=${NGINX_VERSION} \
+    ..
+  cmake --build . -j ${CORES} --target install
+
+  mkdir -p /etc/nginx/modules
+  cp ${INSTAL_DIR}/otel_ngx_module.so /etc/nginx/modules/otel_ngx_module.so
+
+  mkdir -p ${INSTAL_DIR}/lib
+  cp /usr/lib/libopentelemetry_exporter_otlp_grpc.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libopentelemetry_otlp_recordable.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libprotobuf.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libopentelemetry_trace.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libopentelemetry_resources.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libopentelemetry_common.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libstdc++.so* ${INSTAL_DIR}/lib
+
+  cp /usr/lib/libgrpc.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libgcc_s.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libgrpc++.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_bad_variant_access.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_synchronization.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_raw_hash_set.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_hash.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_statusor.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libgpr.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libupb.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_status.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_time.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_strings.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_stacktrace.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_symbolize.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_malloc_internal.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_base.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_spinlock_wait.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_raw_logging_internal.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libre2.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libaddress_sorting.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_cord.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_bad_optional_access.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_str_format_internal.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_throw_delegate.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_time_zone.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_city.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_low_level_hash.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_cordz_info.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_int128.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_strings_internal.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_debugging_internal.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_cord_internal.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_cordz_functions.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_cordz_handle.so* ${INSTAL_DIR}/lib
+  cp /usr/lib/libabsl_exponential_biased.so* ${INSTAL_DIR}/lib
+}
 
-get_src e8d0290ff561986ad7cd6c33307e12e11b137186c4403a6a5ccdb4914c082d88 \
-        "https://nginx.org/download/nginx-$NGINX_VERSION.tar.gz"
-
-get_src 360cdcbd1a235ec62119cc53956b2d31b6ff5f41d44415be53acc544709d58b8 \
-        "https://github.com/open-telemetry/opentelemetry-cpp-contrib/archive/$OPENTELEMETRY_CONTRIB_COMMIT.tar.gz"
-
-# improve compilation times
-CORES=$(($(grep -c ^processor /proc/cpuinfo) - 1))
-
-export MAKEFLAGS=-j${CORES}
-
-apk add \
-  protobuf-dev \
-  grpc \
-  grpc-dev \
-  gtest-dev \
-  c-ares-dev \
-  pcre-dev
-
-cd $BUILD_PATH
-git clone --recursive https://github.com/open-telemetry/opentelemetry-cpp opentelemetry-cpp-$OPENTELEMETRY_CPP_VERSION
-cd "opentelemetry-cpp-$OPENTELEMETRY_CPP_VERSION"
-git checkout v$OPENTELEMETRY_CPP_VERSION
-mkdir .build
-cd .build
-
-cmake -DCMAKE_BUILD_TYPE=Release \
-      -DBUILD_TESTING=OFF \
-      -DWITH_EXAMPLES=OFF \
-      -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
-      -DWITH_OTLP=ON \
-      -DWITH_OTLP_HTTP=OFF \
-      ..
-make
-make install
-
-# build nginx
-cd "$BUILD_PATH/nginx-$NGINX_VERSION"
-./configure \
-  --prefix=/usr/local/nginx \
-  --with-compat \
-  --add-dynamic-module=$BUILD_PATH/opentelemetry-cpp-contrib-$OPENTELEMETRY_CONTRIB_COMMIT/instrumentation/nginx
-
-make modules
-mkdir -p /etc/nginx/modules
-cp objs/otel_ngx_module.so /etc/nginx/modules/otel_ngx_module.so
-
-# remove .a files
-find /usr/local -name "*.a" -print | xargs /bin/rm
+while getopts ":hpng:o:" option; do
+   case $option in
+    h) # display Help
+         Help
+         exit;;
+    g) # install gRPC with git tag
+        GRPC_GIT_TAG=${OPTARG}
+        install_grpc
+        exit;;
+    o) # install OpenTelemetry tag
+        OPENTELEMETRY_CPP_VERSION=${OPTARG}
+        install_otel
+        exit;;
+    p) # prepare
+        prepare
+        exit;;
+    n) # install nginx
+        install_nginx
+        exit;;
+    \?)
+        Help
+        exit;;
+   esac
+done
diff --git a/internal/ingress/annotations/authreq/main.go b/internal/ingress/annotations/authreq/main.go
index 5dd1126fa..b5dc4c70b 100644
--- a/internal/ingress/annotations/authreq/main.go
+++ b/internal/ingress/annotations/authreq/main.go
@@ -48,6 +48,7 @@ type Config struct {
 	KeepaliveRequests      int               `json:"keepaliveRequests"`
 	KeepaliveTimeout       int               `json:"keepaliveTimeout"`
 	ProxySetHeaders        map[string]string `json:"proxySetHeaders,omitempty"`
+	AlwaysSetCookie        bool              `json:"alwaysSetCookie,omitempty"`
 }
 
 // DefaultCacheDuration is the fallback value if no cache duration is provided
@@ -112,6 +113,10 @@ func (e1 *Config) Equal(e2 *Config) bool {
 		return false
 	}
 
+	if e1.AlwaysSetCookie != e2.AlwaysSetCookie {
+		return false
+	}
+
 	return sets.StringElementsMatch(e1.AuthCacheDuration, e2.AuthCacheDuration)
 }
 
@@ -297,6 +302,8 @@ func (a authReq) Parse(ing *networking.Ingress) (interface{}, error) {
 
 	requestRedirect, _ := parser.GetStringAnnotation("auth-request-redirect", ing)
 
+	alwaysSetCookie, _ := parser.GetBoolAnnotation("auth-always-set-cookie", ing)
+
 	return &Config{
 		URL:                    urlString,
 		Host:                   authURL.Hostname(),
@@ -312,6 +319,7 @@ func (a authReq) Parse(ing *networking.Ingress) (interface{}, error) {
 		KeepaliveRequests:      keepaliveRequests,
 		KeepaliveTimeout:       keepaliveTimeout,
 		ProxySetHeaders:        proxySetHeaders,
+		AlwaysSetCookie:        alwaysSetCookie,
 	}, nil
 }
 
diff --git a/internal/ingress/annotations/ipwhitelist/main.go b/internal/ingress/annotations/ipwhitelist/main.go
index 77c2b6cc0..38610ade6 100644
--- a/internal/ingress/annotations/ipwhitelist/main.go
+++ b/internal/ingress/annotations/ipwhitelist/main.go
@@ -62,18 +62,21 @@ func NewParser(r resolver.Resolver) parser.IngressAnnotation {
 // e.g. `18.0.0.0/8,56.0.0.0/8`
 func (a ipwhitelist) Parse(ing *networking.Ingress) (interface{}, error) {
 	defBackend := a.r.GetDefaultBackend()
-	sort.Strings(defBackend.WhitelistSourceRange)
+
+	defaultWhitelistSourceRange := make([]string, len(defBackend.WhitelistSourceRange))
+	copy(defaultWhitelistSourceRange, defBackend.WhitelistSourceRange)
+	sort.Strings(defaultWhitelistSourceRange)
 
 	val, err := parser.GetStringAnnotation("whitelist-source-range", ing)
 	// A missing annotation is not a problem, just use the default
 	if err == ing_errors.ErrMissingAnnotations {
-		return &SourceRange{CIDR: defBackend.WhitelistSourceRange}, nil
+		return &SourceRange{CIDR: defaultWhitelistSourceRange}, nil
 	}
 
 	values := strings.Split(val, ",")
 	ipnets, ips, err := net.ParseIPNets(values...)
 	if err != nil && len(ips) == 0 {
-		return &SourceRange{CIDR: defBackend.WhitelistSourceRange}, ing_errors.LocationDenied{
+		return &SourceRange{CIDR: defaultWhitelistSourceRange}, ing_errors.LocationDenied{
 			Reason: fmt.Errorf("the annotation does not contain a valid IP address or network: %w", err),
 		}
 	}
diff --git a/internal/ingress/controller/config/config.go b/internal/ingress/controller/config/config.go
index f0dfdc191..a1f35e9fa 100644
--- a/internal/ingress/controller/config/config.go
+++ b/internal/ingress/controller/config/config.go
@@ -782,7 +782,7 @@ func NewDefault() Configuration {
 	defNginxStatusIpv4Whitelist = append(defNginxStatusIpv4Whitelist, "127.0.0.1")
 	defNginxStatusIpv6Whitelist = append(defNginxStatusIpv6Whitelist, "::1")
 	defProxyDeadlineDuration := time.Duration(5) * time.Second
-	defGlobalExternalAuth := GlobalExternalAuth{"", "", "", "", "", append(defResponseHeaders, ""), "", "", "", []string{}, map[string]string{}}
+	defGlobalExternalAuth := GlobalExternalAuth{"", "", "", "", "", append(defResponseHeaders, ""), "", "", "", []string{}, map[string]string{}, false}
 
 	cfg := Configuration{
 
@@ -995,4 +995,5 @@ type GlobalExternalAuth struct {
 	AuthCacheKey           string            `json:"authCacheKey"`
 	AuthCacheDuration      []string          `json:"authCacheDuration"`
 	ProxySetHeaders        map[string]string `json:"proxySetHeaders,omitempty"`
+	AlwaysSetCookie        bool              `json:"alwaysSetCookie,omitempty"`
 }
diff --git a/internal/ingress/controller/controller.go b/internal/ingress/controller/controller.go
index f43c72d92..44385f175 100644
--- a/internal/ingress/controller/controller.go
+++ b/internal/ingress/controller/controller.go
@@ -97,9 +97,10 @@ type Configuration struct {
 
 	EnableProfiling bool
 
-	EnableMetrics  bool
-	MetricsPerHost bool
-	MetricsBuckets *collectors.HistogramBuckets
+	EnableMetrics       bool
+	MetricsPerHost      bool
+	MetricsBuckets      *collectors.HistogramBuckets
+	ReportStatusClasses bool
 
 	FakeCertificate *ingress.SSLCert
 
diff --git a/internal/ingress/controller/template/configmap.go b/internal/ingress/controller/template/configmap.go
index 4f3aca444..382cb0367 100644
--- a/internal/ingress/controller/template/configmap.go
+++ b/internal/ingress/controller/template/configmap.go
@@ -62,6 +62,7 @@ const (
 	globalAuthSnippet             = "global-auth-snippet"
 	globalAuthCacheKey            = "global-auth-cache-key"
 	globalAuthCacheDuration       = "global-auth-cache-duration"
+	globalAuthAlwaysSetCookie     = "global-auth-always-set-cookie"
 	luaSharedDictsKey             = "lua-shared-dicts"
 	plugins                       = "plugins"
 )
@@ -315,6 +316,16 @@ func ReadConfig(src map[string]string) config.Configuration {
 		to.GlobalExternalAuth.AuthCacheDuration = cacheDurations
 	}
 
+	if val, ok := conf[globalAuthAlwaysSetCookie]; ok {
+		delete(conf, globalAuthAlwaysSetCookie)
+
+		alwaysSetCookie, err := strconv.ParseBool(val)
+		if err != nil {
+			klog.Warningf("Global auth location denied - %s", fmt.Errorf("cannot convert %s to bool: %v", globalAuthAlwaysSetCookie, err))
+		}
+		to.GlobalExternalAuth.AlwaysSetCookie = alwaysSetCookie
+	}
+
 	// Verify that the configured timeout is parsable as a duration. if not, set the default value
 	if val, ok := conf[proxyHeaderTimeout]; ok {
 		delete(conf, proxyHeaderTimeout)
diff --git a/internal/ingress/metric/collectors/socket.go b/internal/ingress/metric/collectors/socket.go
index ea4775583..26aebbaef 100644
--- a/internal/ingress/metric/collectors/socket.go
+++ b/internal/ingress/metric/collectors/socket.go
@@ -85,7 +85,8 @@ type SocketCollector struct {
 
 	hosts sets.String
 
-	metricsPerHost bool
+	metricsPerHost      bool
+	reportStatusClasses bool
 
 	buckets HistogramBuckets
 }
@@ -110,7 +111,7 @@ var defObjectives = map[float64]float64{0.5: 0.05, 0.9: 0.01, 0.99: 0.001}
 
 // NewSocketCollector creates a new SocketCollector instance using
 // the ingress watch namespace and class used by the controller
-func NewSocketCollector(pod, namespace, class string, metricsPerHost bool, buckets HistogramBuckets) (*SocketCollector, error) {
+func NewSocketCollector(pod, namespace, class string, metricsPerHost, reportStatusClasses bool, buckets HistogramBuckets) (*SocketCollector, error) {
 	socket := "/tmp/nginx/prometheus-nginx.socket"
 	// unix sockets must be unlink()ed before being used
 	_ = syscall.Unlink(socket)
@@ -139,7 +140,8 @@ func NewSocketCollector(pod, namespace, class string, metricsPerHost bool, bucke
 	sc := &SocketCollector{
 		listener: listener,
 
-		metricsPerHost: metricsPerHost,
+		metricsPerHost:      metricsPerHost,
+		reportStatusClasses: reportStatusClasses,
 
 		responseTime: prometheus.NewHistogramVec(
 			prometheus.HistogramOpts{
@@ -249,6 +251,10 @@ func (sc *SocketCollector) handleMessage(msg []byte) {
 			continue
 		}
 
+		if sc.reportStatusClasses && len(stats.Status) > 0 {
+			stats.Status = fmt.Sprintf("%cxx", stats.Status[0])
+		}
+
 		// Note these must match the order in requestTags at the top
 		requestLabels := prometheus.Labels{
 			"status":    stats.Status,
diff --git a/internal/ingress/metric/main.go b/internal/ingress/metric/main.go
index 97a8ca7d4..a0aea1e71 100644
--- a/internal/ingress/metric/main.go
+++ b/internal/ingress/metric/main.go
@@ -69,7 +69,7 @@ type collector struct {
 }
 
 // NewCollector creates a new metric collector the for ingress controller
-func NewCollector(metricsPerHost bool, registry *prometheus.Registry, ingressclass string, buckets collectors.HistogramBuckets) (Collector, error) {
+func NewCollector(metricsPerHost, reportStatusClasses bool, registry *prometheus.Registry, ingressclass string, buckets collectors.HistogramBuckets) (Collector, error) {
 	podNamespace := os.Getenv("POD_NAMESPACE")
 	if podNamespace == "" {
 		podNamespace = "default"
@@ -87,7 +87,7 @@ func NewCollector(metricsPerHost bool, registry *prometheus.Registry, ingresscla
 		return nil, err
 	}
 
-	s, err := collectors.NewSocketCollector(podName, podNamespace, ingressclass, metricsPerHost, buckets)
+	s, err := collectors.NewSocketCollector(podName, podNamespace, ingressclass, metricsPerHost, reportStatusClasses, buckets)
 	if err != nil {
 		return nil, err
 	}
diff --git a/rootfs/etc/nginx/template/nginx.tmpl b/rootfs/etc/nginx/template/nginx.tmpl
index ed7d3cc08..bd0880c77 100755
--- a/rootfs/etc/nginx/template/nginx.tmpl
+++ b/rootfs/etc/nginx/template/nginx.tmpl
@@ -778,7 +778,7 @@ stream {
     access_log {{ or $cfg.StreamAccessLogPath $cfg.AccessLogPath }} log_stream {{ $cfg.AccessLogParams }};
     {{ end }}
 
-   
+
     error_log  {{ $cfg.ErrorLogPath }} {{ $cfg.ErrorLogLevel }};
     {{ if $cfg.EnableRealIp }}
     {{ range $trusted_ip := $cfg.ProxyRealIPCIDR }}
@@ -1292,7 +1292,11 @@ stream {
             {{ else }}
             auth_request        {{ $authPath }};
             auth_request_set    $auth_cookie $upstream_http_set_cookie;
+            {{ if $externalAuth.AlwaysSetCookie }}
+            add_header          Set-Cookie $auth_cookie always;
+            {{ else }}
             add_header          Set-Cookie $auth_cookie;
+            {{ end }}
             {{- range $line := buildAuthResponseHeaders $proxySetHeader $externalAuth.ResponseHeaders false }}
             {{ $line }}
             {{- end }}


Test patch:
diff --git a/docs/e2e-tests.md b/docs/e2e-tests.md
index fa8158887..d5aa6cc16 100644
--- a/docs/e2e-tests.md
+++ b/docs/e2e-tests.md
@@ -1,490 +1,249 @@
-
+<!---
+This file is autogenerated!
+Do not try to edit it manually.
+-->
 
 # e2e test suite for [Ingress NGINX Controller](https://github.com/kubernetes/ingress-nginx/tree/main/)
 
 
 
-### [[Default Backend] change default settings](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/defaultbackend/with_hosts.go#L31)
-
-- [should apply the annotation to the default backend](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/defaultbackend/with_hosts.go#L39)
-
-### [[Default Backend]](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/defaultbackend/default_backend.go#L29)
-
-- [should return 404 sending requests when only a default backend is running](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/defaultbackend/default_backend.go#L32)
-- [enables access logging for default backend](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/defaultbackend/default_backend.go#L89)
-- [disables access logging for default backend](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/defaultbackend/default_backend.go#L106)
-
-### [[Default Backend] custom service](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/defaultbackend/custom_default_backend.go#L33)
-
-- [uses custom default backend that returns 200 as status code](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/defaultbackend/custom_default_backend.go#L36)
-
-### [[Default Backend] SSL](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/defaultbackend/ssl.go#L26)
-
-- [should return a self generated SSL certificate](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/defaultbackend/ssl.go#L29)
-
-### [[TCP] tcp-services](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/tcpudp/tcp.go#L37)
-
-- [should expose a TCP service](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/tcpudp/tcp.go#L40)
-- [should expose an ExternalName TCP service](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/tcpudp/tcp.go#L98)
-
-### [auth-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L39)
-
-- [should return status code 200 when no authentication is configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L46)
-- [should return status code 503 when authentication is configured with an invalid secret](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L65)
-- [should return status code 401 when authentication is configured but Authorization header is not configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L89)
-- [should return status code 401 when authentication is configured and Authorization header is sent with invalid credentials](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L116)
-- [should return status code 200 when authentication is configured and Authorization header is sent](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L144)
-- [should return status code 200 when authentication is configured with a map and Authorization header is sent](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L171)
-- [should return status code 401 when authentication is configured with invalid content and Authorization header is sent](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L199)
-- [proxy_set_header My-Custom-Header 42;](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L238)
-- [proxy_set_header My-Custom-Header 42;](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L256)
-- [proxy_set_header 'My-Custom-Header' '42';](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L273)
-- [](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L294)
-- [retains cookie set by external authentication server](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L316)
-- [should return status code 200 when signed in](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L413)
-- [should redirect to signin url when not signed in](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L422)
-- [keeps processing new ingresses even if one of the existing ingresses is misconfigured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L433)
-- [should return status code 200 when signed in](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L490)
-- [should redirect to signin url when not signed in](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L499)
-- [keeps processing new ingresses even if one of the existing ingresses is misconfigured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L510)
-- [should return status code 200 when signed in after auth backend is deleted ](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L584)
-- [should deny login for different location on same server](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L604)
-- [should deny login for different servers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L632)
-- [should redirect to signin url when not signed in](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/auth.go#L661)
-
-### [affinitymode](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinitymode.go#L31)
-
-- [Balanced affinity mode should balance](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinitymode.go#L34)
-- [Check persistent affinity mode](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinitymode.go#L64)
-
-### [proxy-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L28)
-
-- [should set proxy_redirect to off](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L36)
-- [should set proxy_redirect to default](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L52)
-- [should set proxy_redirect to hello.com goodbye.com](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L68)
-- [should set proxy client-max-body-size to 8m](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L85)
-- [should not set proxy client-max-body-size to incorrect value](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L100)
-- [should set valid proxy timeouts](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L115)
-- [should not set invalid proxy timeouts](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L136)
-- [should turn on proxy-buffering](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L157)
-- [should turn off proxy-request-buffering](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L179)
-- [should build proxy next upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L194)
-- [should setup proxy cookies](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L215)
-- [should change the default proxy HTTP version](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxy.go#L233)
-
-### [affinity session-cookie-name](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L35)
-
-- [should set sticky cookie SERVERID](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L42)
-- [should change cookie name on ingress definition change](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L64)
-- [should set the path to /something on the generated cookie](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L99)
-- [does not set the path to / on the generated cookie if there's more than one rule referring to the same backend](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L121)
-- [should set cookie with expires](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L182)
-- [should work with use-regex annotation and session-cookie-path](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L213)
-- [should warn user when use-regex is true and session-cookie-path is not set](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L237)
-- [should not set affinity across all server locations when using separate ingresses](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L263)
-- [should set sticky cookie without host](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L295)
-- [should work with server-alias annotation](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/affinity.go#L315)
-
-### [mirror-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/mirror.go#L28)
-
-- [should set mirror-target to http://localhost/mirror](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/mirror.go#L36)
-- [should set mirror-target to https://test.env.com/$request_uri](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/mirror.go#L51)
-- [should disable mirror-request-body](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/mirror.go#L67)
-
-### [canary-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L35)
-
-- [should response with a 200 status from the mainline upstream when requests are made to the mainline ingress](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L47)
-- [should return 404 status for requests to the canary if no matching ingress is found](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L79)
-- [should return the correct status codes when endpoints are unavailable](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L106)
-- [should route requests to the correct upstream if mainline ingress is created before the canary ingress](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L160)
-- [should route requests to the correct upstream if mainline ingress is created after the canary ingress](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L205)
-- [should route requests to the correct upstream if the mainline ingress is modified](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L249)
-- [should route requests to the correct upstream if the canary ingress is modified](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L306)
-- [should route requests to the correct upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L361)
-- [should route requests to the correct upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L415)
-- [should route requests to the correct upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L479)
-- [should route requests to the correct upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L521)
-- [should routes to mainline upstream when the given Regex causes error](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L555)
-- [should route requests to the correct upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L593)
-- [should route requests to the correct upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L632)
-- [should route requests to the correct upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L687)
-- [should not use canary as a catch-all server](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L743)
-- [should not use canary with domain as a server](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L771)
-- [does not crash when canary ingress has multiple paths to the same non-matching backend](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/canary.go#L795)
-
-### [limit-rate](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/limitrate.go#L29)
-
-- [Check limit-rate annotation](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/limitrate.go#L37)
-
-### [force-ssl-redirect](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/forcesslredirect.go#L27)
-
-- [should redirect to https](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/forcesslredirect.go#L34)
-
-### [http2-push-preload](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/http2pushpreload.go#L27)
-
-- [enable the http2-push-preload directive](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/http2pushpreload.go#L34)
-
-### [proxy-ssl-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxyssl.go#L30)
-
-- [should set valid proxy-ssl-secret](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxyssl.go#L37)
-- [should set valid proxy-ssl-secret, proxy-ssl-verify to on, proxy-ssl-verify-depth to 2, and proxy-ssl-server-name to on](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxyssl.go#L64)
-- [should set valid proxy-ssl-secret, proxy-ssl-ciphers to HIGH:!AES](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxyssl.go#L94)
-- [should set valid proxy-ssl-secret, proxy-ssl-protocols](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxyssl.go#L122)
-- [proxy-ssl-location-only flag should change the nginx config server part](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/proxyssl.go#L150)
-
-### [modsecurity owasp](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/modsecurity/modsecurity.go#L27)
-
-- [should enable modsecurity](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/modsecurity/modsecurity.go#L34)
-- [should enable modsecurity with transaction ID and OWASP rules](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/modsecurity/modsecurity.go#L52)
-- [should disable modsecurity](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/modsecurity/modsecurity.go#L73)
-- [should enable modsecurity with snippet](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/modsecurity/modsecurity.go#L90)
-- [should enable modsecurity without using 'modsecurity on;'](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/modsecurity/modsecurity.go#L109)
-- [should disable modsecurity using 'modsecurity off;'](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/modsecurity/modsecurity.go#L131)
-- [should enable modsecurity with snippet and block requests](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/modsecurity/modsecurity.go#L152)
-- [should enable modsecurity globally and with modsecurity-snippet block requests](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/modsecurity/modsecurity.go#L186)
-
-### [backend-protocol - GRPC](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/grpc.go#L38)
-
-- [should use grpc_pass in the configuration file](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/grpc.go#L41)
-- [should return OK for service with backend protocol GRPC](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/grpc.go#L66)
-- [should return OK for service with backend protocol GRPCS](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/grpc.go#L124)
-
-### [cors-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/cors.go#L28)
-
-- [should enable cors](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/cors.go#L35)
-- [should set cors methods to only allow POST, GET](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/cors.go#L60)
-- [should set cors max-age](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/cors.go#L76)
-- [should disable cors allow credentials](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/cors.go#L92)
-- [should allow origin for cors](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/cors.go#L108)
-- [should allow headers for cors](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/cors.go#L124)
-- [should expose headers for cors](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/cors.go#L140)
-
-### [influxdb-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/influxdb.go#L39)
-
-- [should send the request metric to the influxdb server](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/influxdb.go#L48)
-
-### [Annotation - limit-connections](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/limitconnections.go#L31)
+### [Geoip2](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/geoip2.go#L37)
 
-- [should limit-connections](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/limitconnections.go#L38)
+- [should include geoip2 line in config when enabled and db file exists](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/geoip2.go#L46)
+- [should only allow requests from specific countries](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/geoip2.go#L70)
 
-### [client-body-buffer-size](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/clientbodybuffersize.go#L28)
+### [[Security] global-auth-url](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L34)
 
-- [should set client_body_buffer_size to 1000](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/clientbodybuffersize.go#L35)
-- [should set client_body_buffer_size to 1K](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/clientbodybuffersize.go#L51)
-- [should set client_body_buffer_size to 1k](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/clientbodybuffersize.go#L67)
-- [should set client_body_buffer_size to 1m](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/clientbodybuffersize.go#L83)
-- [should set client_body_buffer_size to 1M](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/clientbodybuffersize.go#L99)
-- [should not set client_body_buffer_size to invalid 1b](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/clientbodybuffersize.go#L115)
+- [should return status code 401 when request any protected service](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L85)
+- [should return status code 200 when request whitelisted (via no-auth-locations) service and 401 when request protected service](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L102)
+- [should return status code 200 when request whitelisted (via ingress annotation) service and 401 when request protected service](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L126)
+- [should still return status code 200 after auth backend is deleted using cache](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L155)
+- [should proxy_method method when global-auth-method is configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L197)
+- [should add custom error page when global-auth-signin url is configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L210)
+- [should add auth headers when global-auth-response-headers is configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L223)
+- [should set request-redirect when global-auth-request-redirect is configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L237)
+- [should set snippet when global external auth is configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L250)
+- [user retains cookie by default](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L326)
+- [user does not retain cookie if upstream returns error status code](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L337)
+- [user with global-auth-always-set-cookie key in configmap retains cookie if upstream returns error status code](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L348)
 
-### [default-backend](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/default_backend.go#L29)
-
-- [should use a custom default backend as upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/default_backend.go#L37)
-
-### [connection-proxy-header](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/connection.go#L29)
-
-- [set connection header to keep-alive](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/connection.go#L36)
-
-### [upstream-vhost](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/upstreamvhost.go#L27)
-
-- [set host to upstreamvhost.bar.com](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/upstreamvhost.go#L34)
-
-### [custom-http-errors](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/customhttperrors.go#L34)
-
-- [configures Nginx correctly](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/customhttperrors.go#L41)
-
-### [disable-access-log disable-http-access-log disable-stream-access-log](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/disableaccesslog.go#L27)
-
-- [disable-access-log set access_log off](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/disableaccesslog.go#L34)
-- [disable-http-access-log set access_log off](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/disableaccesslog.go#L46)
-
-### [server-snippet](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/serversnippet.go#L27)
-
-- [](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/serversnippet.go#L34)
-
-### [rewrite-target use-regex enable-rewrite-log](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/rewrite.go#L30)
-
-- [should write rewrite logs](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/rewrite.go#L37)
-- [should use correct longest path match](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/rewrite.go#L66)
-- [should use ~* location modifier if regex annotation is present](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/rewrite.go#L111)
-- [should fail to use longest match for documented warning](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/rewrite.go#L158)
-- [should allow for custom rewrite parameters](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/rewrite.go#L190)
-
-### [app-root](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/approot.go#L28)
-
-- [should redirect to /foo](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/approot.go#L35)
-
-### [whitelist-source-range](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/ipwhitelist.go#L26)
-
-- [should set valid ip whitelist range](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/ipwhitelist.go#L33)
-
-### [enable-access-log enable-rewrite-log](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/log.go#L27)
+### [[Security] Pod Security Policies](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/pod_security_policy.go#L40)
 
-- [set access_log off](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/log.go#L34)
-- [set rewrite_log on](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/log.go#L49)
+- [should be running with a Pod Security Policy](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/pod_security_policy.go#L43)
 
-### [x-forwarded-prefix](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/xforwardedprefix.go#L28)
+### [log-format-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L28)
 
-- [should set the X-Forwarded-Prefix to the annotation value](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/xforwardedprefix.go#L35)
-- [should not add X-Forwarded-Prefix if the annotation value is empty](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/xforwardedprefix.go#L57)
+- [should disable the log-format-escape-json by default](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L40)
+- [should enable the log-format-escape-json](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L47)
+- [should disable the log-format-escape-json](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L55)
+- [log-format-escape-json enabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L66)
+- [log-format-escape-json disabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L89)
 
-### [configuration-snippet](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/snippet.go#L27)
+### [server-tokens](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/server_tokens.go#L29)
 
-- [ in all locations](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/snippet.go#L34)
+- [should not exists Server header in the response](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/server_tokens.go#L38)
+- [should exists Server header in the response when is enabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/server_tokens.go#L50)
 
-### [backend-protocol - FastCGI](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/fastcgi.go#L31)
+### [proxy-connect-timeout](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_connect_timeout.go#L28)
 
-- [should use fastcgi_pass in the configuration file](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/fastcgi.go#L38)
-- [should add fastcgi_index in the configuration file](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/fastcgi.go#L55)
-- [should add fastcgi_param in the configuration file](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/fastcgi.go#L72)
-- [should return OK for service with backend protocol FastCGI](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/fastcgi.go#L105)
+- [should set valid proxy timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_connect_timeout.go#L36)
+- [should not set invalid proxy timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_connect_timeout.go#L52)
 
-### [from-to-www-redirect](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/fromtowwwredirect.go#L31)
+### [ssl-ciphers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ssl_ciphers.go#L28)
 
-- [should redirect from www HTTP to HTTP](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/fromtowwwredirect.go#L38)
-- [should redirect from www HTTPS to HTTPS](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/fromtowwwredirect.go#L64)
+- [Add ssl ciphers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ssl_ciphers.go#L31)
 
-### [permanent-redirect permanent-redirect-code](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/redirect.go#L30)
+### [use-proxy-protocol](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_protocol.go#L36)
 
-- [should respond with a standard redirect code](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/redirect.go#L33)
-- [should respond with a custom redirect code](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/redirect.go#L61)
+- [should respect port passed by the PROXY Protocol](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_protocol.go#L46)
+- [should respect proto passed by the PROXY Protocol server port](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_protocol.go#L79)
+- [should enable PROXY Protocol for HTTPS](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_protocol.go#L112)
+- [should enable PROXY Protocol for TCP](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_protocol.go#L155)
 
-### [upstream-hash-by-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/upstreamhashby.go#L76)
+### [plugins](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/plugins.go#L28)
 
-- [should connect to the same pod](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/upstreamhashby.go#L83)
-- [should connect to the same subset of pods](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/upstreamhashby.go#L92)
+- [should exist a x-hello-world header](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/plugins.go#L35)
 
-### [annotation-global-rate-limit](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/globalratelimit.go#L30)
+### [configmap server-snippet](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/server_snippet.go#L28)
 
-- [generates correct configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/globalratelimit.go#L38)
+- [should add value of server-snippet setting to all ingress config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/server_snippet.go#L35)
+- [should add global server-snippet and drop annotations per admin config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/server_snippet.go#L92)
 
-### [backend-protocol](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/backendprotocol.go#L27)
+### [[Flag] disable-catch-all](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_catch_all.go#L33)
 
-- [should set backend protocol to https:// and use proxy_pass](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/backendprotocol.go#L34)
-- [should set backend protocol to grpc:// and use grpc_pass](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/backendprotocol.go#L49)
-- [should set backend protocol to grpcs:// and use grpc_pass](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/backendprotocol.go#L64)
-- [should set backend protocol to '' and use fastcgi_pass](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/backendprotocol.go#L79)
-- [should set backend protocol to '' and use ajp_pass](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/backendprotocol.go#L94)
+- [should ignore catch all Ingress with backend](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_catch_all.go#L50)
+- [should ignore catch all Ingress with backend and rules](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_catch_all.go#L69)
+- [should delete Ingress updated to catch-all](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_catch_all.go#L81)
+- [should allow Ingress with rules](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_catch_all.go#L123)
 
-### [satisfy](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/satisfy.go#L35)
+### [enable-real-ip](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/enable_real_ip.go#L30)
 
-- [should configure satisfy directive correctly](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/satisfy.go#L42)
-- [should allow multiple auth with satisfy any](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/satisfy.go#L84)
+- [trusts X-Forwarded-For header only when setting is true](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/enable_real_ip.go#L40)
+- [should not trust X-Forwarded-For header when setting is false](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/enable_real_ip.go#L78)
 
-### [server-alias](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/alias.go#L29)
+### [keep-alive keep-alive-requests](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L28)
 
-- [should return status code 200 for host 'foo' and 404 for 'bar'](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/alias.go#L36)
-- [should return status code 200 for host 'foo' and 'bar'](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/alias.go#L62)
-- [should return status code 200 for hosts defined in two ingresses, different path with one alias](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/alias.go#L87)
+- [should set keepalive_timeout](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L40)
+- [should set keepalive_requests](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L48)
+- [should set keepalive connection to upstream server](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L59)
+- [should set keep alive connection timeout to upstream server](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L68)
+- [should set the request count to upstream server through one keep alive connection](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L77)
 
-### [ssl-ciphers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/sslciphers.go#L27)
+### [enable-multi-accept](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/multi_accept.go#L27)
 
-- [should change ssl ciphers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/sslciphers.go#L34)
+- [should be enabled by default](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/multi_accept.go#L31)
+- [should be enabled when set to true](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/multi_accept.go#L39)
+- [should be disabled when set to false](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/multi_accept.go#L49)
 
-### [auth-tls-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/authtls.go#L30)
+### [[Flag] watch namespace selector](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/namespace_selector.go#L30)
 
-- [should set valid auth-tls-secret](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/authtls.go#L37)
-- [should set valid auth-tls-secret, sslVerify to off, and sslVerifyDepth to 2](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/authtls.go#L73)
-- [should set valid auth-tls-secret, pass certificate to upstream, and error page](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/authtls.go#L103)
-- [should validate auth-tls-verify-client](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/annotations/authtls.go#L153)
+- [should ingore Ingress of namespace without label foo=bar and accept those of namespace with label foo=bar](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/namespace_selector.go#L70)
 
-### [[Status] status update](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/status/update.go#L38)
+### [[Flag] disable-service-external-name](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_service_external_name.go#L34)
 
-- [should update status field after client-go reconnection](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/status/update.go#L43)
+- [should ignore services of external-name type](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_service_external_name.go#L51)
 
-### [Debug CLI](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/dbg/main.go#L29)
+### [Configure OpenTracing](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L48)
 
-- [should list the backend servers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/dbg/main.go#L37)
-- [should get information for a specific backend server](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/dbg/main.go#L56)
-- [should produce valid JSON for /dbg general](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/dbg/main.go#L85)
+- [should not exists opentracing directive](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L58)
+- [should exists opentracing directive when is enabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L71)
+- [should include opentracing_trust_incoming_span off directive when disabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L85)
+- [should not exists opentracing_operation_name directive when is empty](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L100)
+- [should exists opentracing_operation_name directive when is configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L115)
+- [should not exists opentracing_location_operation_name directive when is empty](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L130)
+- [should exists opentracing_location_operation_name directive when is configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L145)
+- [should enable opentracing using zipkin](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L160)
+- [should enable opentracing using jaeger](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L172)
+- [should enable opentracing using jaeger with sampler host](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L184)
+- [should propagate the w3c header when configured with jaeger](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L197)
+- [should enable opentracing using jaeger with an HTTP endpoint](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L228)
+- [should enable opentracing using datadog](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L241)
 
-### [[Memory Leak] Dynamic Certificates](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/leaks/lua_ssl.go#L35)
+### [Configmap change](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/configmap_change.go#L29)
 
-- [should not leak memory from ingress SSL certificates or configuration updates](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/leaks/lua_ssl.go#L42)
+- [should reload after an update in the configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/configmap_change.go#L36)
 
-### [[Ingress] [PathType] mix Exact and Prefix paths](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/pathtype_mixed.go#L30)
+### [Configmap - limit-rate](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/limit_rate.go#L28)
 
-- [should choose the correct location](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/pathtype_mixed.go#L39)
+- [Check limit-rate config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/limit_rate.go#L36)
 
-### [[Ingress] definition without host](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/without_host.go#L32)
+### [access-log](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L26)
 
-- [should set ingress details variables for ingresses without a host](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/without_host.go#L35)
-- [should set ingress details variables for ingresses with host without IngressRuleValue, only Backend](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/without_host.go#L56)
+- [use the default configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L31)
+- [use the specified configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L41)
+- [use the specified configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L53)
+- [use the specified configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L66)
+- [use the specified configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L79)
 
-### [single ingress - multiple hosts](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/multiple_rules.go#L31)
+### [global-options](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_options.go#L28)
 
-- [should set the correct $service_name NGINX variable](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/multiple_rules.go#L39)
+- [should have worker_rlimit_nofile option](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_options.go#L31)
+- [should have worker_rlimit_nofile option and be independent on amount of worker processes](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_options.go#L38)
 
-### [[Ingress] [PathType] exact](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/pathtype_exact.go#L30)
+### [[Flag] ingress-class](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L39)
 
-- [should choose exact location for /exact](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/pathtype_exact.go#L37)
+- [should ignore Ingress with a different class annotation](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L68)
+- [should ignore Ingress with different controller class](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L104)
+- [should accept both Ingresses with default IngressClassName and IngressClass annotation](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L132)
+- [should ignore Ingress without IngressClass configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L164)
+- [should delete Ingress when class is removed](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L192)
+- [should serve Ingress when class is added](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L257)
+- [should serve Ingress when class is updated between annotation and ingressClassName](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L323)
+- [should ignore Ingress with no class and accept the correctly configured Ingresses](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L413)
+- [should watch Ingress with no class and ignore ingress with a different class](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L482)
+- [should watch Ingress that uses the class name even if spec is different](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L539)
+- [should watch Ingress with correct annotation](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L631)
+- [should ignore Ingress with only IngressClassName](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L652)
 
-### [[Ingress] [PathType] prefix checks](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/pathtype_prefix.go#L28)
+### [reuse-port](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/reuse-port.go#L27)
 
-- [should return 404 when prefix /aaa does not match request /aaaccc](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ingress/pathtype_prefix.go#L35)
+- [reuse port should be enabled by default](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/reuse-port.go#L38)
+- [reuse port should be disabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/reuse-port.go#L44)
+- [reuse port should be enabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/reuse-port.go#L52)
 
-### [[Security] request smuggling](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/security/request_smuggling.go#L32)
+### [proxy-send-timeout](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_send_timeout.go#L28)
 
-- [should not return body content from error_page](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/security/request_smuggling.go#L39)
+- [should set valid proxy send timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_send_timeout.go#L36)
+- [should not set invalid proxy send timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_send_timeout.go#L52)
 
 ### [[SSL] [Flag] default-ssl-certificate](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/default_ssl_certificate.go#L33)
 
 - [uses default ssl certificate for catch-all ingress](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/default_ssl_certificate.go#L64)
 - [uses default ssl certificate for host based ingress when configured certificate does not match host](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/default_ssl_certificate.go#L80)
 
-### [enable-real-ip](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/enable_real_ip.go#L30)
-
-- [trusts X-Forwarded-For header only when setting is true](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/enable_real_ip.go#L40)
-- [should not trust X-Forwarded-For header when setting is false](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/enable_real_ip.go#L78)
-
-### [access-log](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L26)
-
-- [use the default configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L31)
-- [use the specified configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L39)
-- [use the specified configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L51)
-- [use the specified configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L63)
-- [use the specified configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/access_log.go#L75)
-
-### [[Lua] lua-shared-dicts](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/lua_shared_dicts.go#L26)
-
-- [configures lua shared dicts](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/lua_shared_dicts.go#L29)
-
-### [server-tokens](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/server_tokens.go#L30)
+### [brotli](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/brotli.go#L30)
 
-- [should not exists Server header in the response](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/server_tokens.go#L38)
-- [should exists Server header in the response when is enabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/server_tokens.go#L50)
-
-### [use-proxy-protocol](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_protocol.go#L36)
-
-- [should respect port passed by the PROXY Protocol](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_protocol.go#L46)
-- [should respect proto passed by the PROXY Protocol server port](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_protocol.go#L79)
-- [should enable PROXY Protocol for HTTPS](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_protocol.go#L112)
-- [should enable PROXY Protocol for TCP](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_protocol.go#L155)
-
-### [[Flag] custom HTTP and HTTPS ports](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/listen_nondefault_ports.go#L32)
-
-- [should set X-Forwarded-Port headers accordingly when listening on a non-default HTTP port](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/listen_nondefault_ports.go#L48)
-- [should set X-Forwarded-Port header to 443](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/listen_nondefault_ports.go#L70)
-- [should set the X-Forwarded-Port header to 443](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/listen_nondefault_ports.go#L100)
+- [ condition](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/brotli.go#L39)
 
-### [[Security] no-auth-locations](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_auth_locations.go#L34)
+### [[Security] Pod Security Policies with volumes](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/pod_security_policy_volumes.go#L36)
 
-- [should return status code 401 when accessing '/' unauthentication](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_auth_locations.go#L55)
-- [should return status code 200 when accessing '/' authentication](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_auth_locations.go#L69)
-- [should return status code 200 when accessing '/noauth' unauthenticated](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_auth_locations.go#L83)
+- [should be running with a Pod Security Policy](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/pod_security_policy_volumes.go#L39)
 
 ### [Dynamic $proxy_host](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_host.go#L28)
 
 - [should exist a proxy_host](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_host.go#L36)
 - [should exist a proxy_host using the upstream-vhost annotation value](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_host.go#L57)
 
-### [proxy-connect-timeout](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_connect_timeout.go#L28)
-
-- [should set valid proxy timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_connect_timeout.go#L36)
-- [should not set invalid proxy timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_connect_timeout.go#L52)
-
-### [[Security] Pod Security Policies](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/pod_security_policy.go#L40)
-
-- [should be running with a Pod Security Policy](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/pod_security_policy.go#L43)
-
-### [Geoip2](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/geoip2.go#L29)
-
-- [should only allow requests from specific countries](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/geoip2.go#L38)
-
-### [[Security] Pod Security Policies with volumes](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/pod_security_policy_volumes.go#L36)
-
-- [should be running with a Pod Security Policy](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/pod_security_policy_volumes.go#L39)
-
-### [enable-multi-accept](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/multi_accept.go#L27)
-
-- [should be enabled by default](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/multi_accept.go#L31)
-- [should be enabled when set to true](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/multi_accept.go#L39)
-- [should be disabled when set to false](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/multi_accept.go#L49)
-
-### [log-format-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L28)
-
-- [should disable the log-format-escape-json by default](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L40)
-- [should enable the log-format-escape-json](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L47)
-- [should disable the log-format-escape-json](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L55)
-- [log-format-escape-json enabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L66)
-- [log-format-escape-json disabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/log-format.go#L89)
-
-### [[Flag] ingress-class](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L39)
-
-- [should ignore Ingress with class](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L91)
-- [should ignore Ingress with no class](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L143)
-- [should delete Ingress when class is removed](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L177)
-- [check scenarios for IngressClass and ingress.class annotation](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ingress_class.go#L216)
-
-### [ssl-ciphers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ssl_ciphers.go#L28)
+### [[Security] block-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_access_block.go#L28)
 
-- [Add ssl ciphers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ssl_ciphers.go#L31)
+- [should block CIDRs defined in the ConfigMap](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_access_block.go#L38)
+- [should block User-Agents defined in the ConfigMap](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_access_block.go#L55)
+- [should block Referers defined in the ConfigMap](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_access_block.go#L88)
 
-### [proxy-next-upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_next_upstream.go#L28)
+### [settings-global-rate-limit](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/globalratelimit.go#L30)
 
-- [should build proxy next upstream using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_next_upstream.go#L36)
+- [generates correct NGINX configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/globalratelimit.go#L38)
 
-### [[Security] global-auth-url](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L32)
+### [Add no tls redirect locations](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_tls_redirect_locations.go#L28)
 
-- [should return status code 401 when request any protected service](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L83)
-- [should return status code 200 when request whitelisted (via no-auth-locations) service and 401 when request protected service](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L100)
-- [should return status code 200 when request whitelisted (via ingress annotation) service and 401 when request protected service](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L124)
-- [should still return status code 200 after auth backend is deleted using cache ](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L153)
-- [](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L195)
-- [](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L208)
-- [](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L221)
-- [](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L235)
-- [](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_external_auth.go#L248)
+- [Check no tls redirect locations config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_tls_redirect_locations.go#L31)
 
-### [[Security] block-*](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_access_block.go#L28)
+### [main-snippet](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/main_snippet.go#L27)
 
-- [should block CIDRs defined in the ConfigMap](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_access_block.go#L38)
-- [should block User-Agents defined in the ConfigMap](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_access_block.go#L55)
-- [should block Referers defined in the ConfigMap](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/global_access_block.go#L88)
+- [should add value of main-snippet setting to nginx config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/main_snippet.go#L31)
 
-### [plugins](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/plugins.go#L28)
+### [[Lua] lua-shared-dicts](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/lua_shared_dicts.go#L26)
 
-- [should exist a x-hello-world header](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/plugins.go#L35)
+- [configures lua shared dicts](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/lua_shared_dicts.go#L29)
 
-### [Configmap - limit-rate](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/limit_rate.go#L28)
+### [Bad annotation values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/badannotationvalues.go#L29)
 
-- [Check limit-rate config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/limit_rate.go#L36)
+- [[BAD_ANNOTATIONS] should drop an ingress if there is an invalid character in some annotation](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/badannotationvalues.go#L36)
+- [[BAD_ANNOTATIONS] should drop an ingress if there is a forbidden word in some annotation](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/badannotationvalues.go#L67)
+- [[BAD_ANNOTATIONS] should allow an ingress if there is a default blocklist config in place](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/badannotationvalues.go#L102)
+- [[BAD_ANNOTATIONS] should drop an ingress if there is a custom blocklist config in place and allow others to pass](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/badannotationvalues.go#L133)
 
-### [Configure OpenTracing](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L47)
+### [[SSL] TLS protocols, ciphers and headers)](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L31)
 
-- [should not exists opentracing directive](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L57)
-- [should exists opentracing directive when is enabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L70)
-- [should not exists opentracing_operation_name directive when is empty](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L84)
-- [should exists opentracing_operation_name directive when is configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L99)
-- [should not exists opentracing_location_operation_name directive when is empty](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L114)
-- [should exists opentracing_location_operation_name directive when is configured](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L129)
-- [should enable opentracing using zipkin](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L144)
-- [should enable opentracing using jaeger](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L156)
-- [should enable opentracing using jaeger with sampler host](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L168)
-- [should propagate the w3c header when configured with jaeger](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L181)
-- [should enable opentracing using datadog](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/opentracing.go#L225)
+- [setting cipher suite](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L65)
+- [enforcing TLS v1.0](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L87)
+- [setting max-age parameter](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L133)
+- [setting includeSubDomains parameter](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L149)
+- [setting preload parameter](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L168)
+- [overriding what's set from the upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L188)
+- [should not use ports during the HTTP to HTTPS redirection](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L209)
+- [should not use ports or X-Forwarded-Host during the HTTP to HTTPS redirection](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L227)
 
-### [use-forwarded-headers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/forwarded_headers.go#L30)
+### [[Security] modsecurity-snippet](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/modsecurity/modsecurity_snippet.go#L27)
 
-- [should trust X-Forwarded headers when setting is true](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/forwarded_headers.go#L40)
-- [should not trust X-Forwarded headers when setting is false](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/forwarded_headers.go#L90)
+- [should add value of modsecurity-snippet setting to nginx config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/modsecurity/modsecurity_snippet.go#L30)
 
-### [proxy-send-timeout](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_send_timeout.go#L28)
+### [OCSP](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ocsp/ocsp.go#L42)
 
-- [should set valid proxy send timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_send_timeout.go#L36)
-- [should not set invalid proxy send timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_send_timeout.go#L52)
+- [should enable OCSP and contain stapling information in the connection](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ocsp/ocsp.go#L49)
 
-### [Add no tls redirect locations](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_tls_redirect_locations.go#L28)
+### [configmap stream-snippet](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/stream_snippet.go#L34)
 
-- [Check no tls redirect locations config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_tls_redirect_locations.go#L31)
+- [should add value of stream-snippet via config map to nginx config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/stream_snippet.go#L41)
 
-### [settings-global-rate-limit](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/globalratelimit.go#L30)
+### [proxy-read-timeout](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_read_timeout.go#L28)
 
-- [generates correct NGINX configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/globalratelimit.go#L38)
+- [should set valid proxy read timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_read_timeout.go#L36)
+- [should not set invalid proxy read timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_read_timeout.go#L52)
 
-### [add-headers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/custom_header.go#L30)
+### [proxy-next-upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_next_upstream.go#L28)
 
-- [Add a custom header](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/custom_header.go#L40)
-- [Add multiple custom headers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/custom_header.go#L65)
+- [should build proxy next upstream using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_next_upstream.go#L36)
 
 ### [hash size](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/hash-size.go#L27)
 
@@ -496,79 +255,65 @@
 - [should set variables-hash-max-size](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/hash-size.go#L88)
 - [should set vmap-hash-bucket-size](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/hash-size.go#L100)
 
-### [keep-alive keep-alive-requests](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L28)
-
-- [should set keepalive_timeout](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L40)
-- [should set keepalive_requests](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L48)
-- [should set keepalive connection to upstream server](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L59)
-- [should set keep alive connection timeout to upstream server](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L68)
-- [should set the request count to upstream server through one keep alive connection](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/keep-alive.go#L77)
-
-### [[Flag] disable-catch-all](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_catch_all.go#L34)
+### [[Security] no-auth-locations](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_auth_locations.go#L33)
 
-- [should ignore catch all Ingress](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_catch_all.go#L51)
-- [should delete Ingress updated to catch-all](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_catch_all.go#L70)
-- [should allow Ingress with both a default backend and rules](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/disable_catch_all.go#L108)
+- [should return status code 401 when accessing '/' unauthentication](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_auth_locations.go#L54)
+- [should return status code 200 when accessing '/' authentication](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_auth_locations.go#L68)
+- [should return status code 200 when accessing '/noauth' unauthenticated](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/no_auth_locations.go#L82)
 
-### [main-snippet](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/main_snippet.go#L27)
+### [[Flag] custom HTTP and HTTPS ports](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/listen_nondefault_ports.go#L32)
 
-- [should add value of main-snippet setting to nginx config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/main_snippet.go#L31)
+- [should set X-Forwarded-Port headers accordingly when listening on a non-default HTTP port](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/listen_nondefault_ports.go#L48)
+- [should set X-Forwarded-Port header to 443](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/listen_nondefault_ports.go#L70)
+- [should set the X-Forwarded-Port header to 443](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/listen_nondefault_ports.go#L100)
 
-### [[SSL] TLS protocols, ciphers and headers)](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L31)
+### [use-forwarded-headers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/forwarded_headers.go#L30)
 
-- [setting cipher suite](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L65)
-- [enforcing TLS v1.0](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L87)
-- [setting max-age parameter](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L133)
-- [setting includeSubDomains parameter](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L149)
-- [setting preload parameter](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L168)
-- [overriding what's set from the upstream](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L188)
-- [should not use ports during the HTTP to HTTPS redirection](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L209)
-- [should not use ports or X-Forwarded-Host during the HTTP to HTTPS redirection](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/tls.go#L227)
+- [should trust X-Forwarded headers when setting is true](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/forwarded_headers.go#L40)
+- [should not trust X-Forwarded headers when setting is false](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/forwarded_headers.go#L92)
 
-### [Configmap change](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/configmap_change.go#L29)
+### [add-headers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/custom_header.go#L30)
 
-- [should reload after an update in the configuration](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/configmap_change.go#L36)
+- [Add a custom header](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/custom_header.go#L40)
+- [Add multiple custom headers](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/custom_header.go#L65)
 
-### [proxy-read-timeout](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_read_timeout.go#L28)
+### [[Load Balancer] round-robin](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/loadbalance/round_robin.go#L31)
 
-- [should set valid proxy read timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_read_timeout.go#L36)
-- [should not set invalid proxy read timeouts using configmap values](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/proxy_read_timeout.go#L52)
+- [should evenly distribute requests with round-robin (default algorithm)](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/loadbalance/round_robin.go#L39)
 
-### [[Security] modsecurity-snippet](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/modsecurity/modsecurity_snippet.go#L27)
+### [[Load Balancer] EWMA](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/loadbalance/ewma.go#L31)
 
-- [should add value of modsecurity-snippet setting to nginx config](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/modsecurity/modsecurity_snippet.go#L30)
+- [does not fail requests](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/loadbalance/ewma.go#L42)
 
-### [OCSP](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ocsp/ocsp.go#L42)
+### [[Load Balancer] load-balance](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/loadbalance/configmap.go#L28)
 
-- [should enable OCSP and contain stapling information in the connection](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/ocsp/ocsp.go#L49)
+- [should apply the configmap load-balance setting](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/loadbalance/configmap.go#L35)
 
-### [reuse-port](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/reuse-port.go#L27)
+### [[SSL] redirect to HTTPS](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ssl/http_redirect.go#L29)
 
-- [reuse port should be enabled by default](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/reuse-port.go#L38)
-- [reuse port should be disabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/reuse-port.go#L44)
-- [reuse port should be enabled](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/settings/reuse-port.go#L52)
+- [should redirect from HTTP to HTTPS when secret is missing](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ssl/http_redirect.go#L36)
 
-### [[Shutdown] Graceful shutdown with pending request](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/gracefulshutdown/slow_requests.go#L28)
+### [[SSL] secret update](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ssl/secret_update.go#L33)
 
-- [should let slow requests finish before shutting down](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/gracefulshutdown/slow_requests.go#L36)
+- [should not appear references to secret updates not used in ingress rules](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ssl/secret_update.go#L40)
+- [should return the fake SSL certificate if the secret is invalid](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/ssl/secret_update.go#L82)
 
-### [[Shutdown] ingress controller](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/gracefulshutdown/shutdown.go#L33)
+### [[Service] Type ExternalName](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L58)
 
-- [should shutdown in less than 60 secons without pending connections](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/gracefulshutdown/shutdown.go#L43)
-- [should shutdown after waiting 60 seconds for pending connections to be closed](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/gracefulshutdown/shutdown.go#L64)
-- [should shutdown after waiting 150 seconds for pending connections to be closed](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/gracefulshutdown/shutdown.go#L109)
+- [works with external name set to incomplete fqdn](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L61)
+- [should return 200 for service type=ExternalName without a port defined](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L94)
+- [should return 200 for service type=ExternalName with a port defined](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L128)
+- [should return status 502 for service type=ExternalName with an invalid host](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L152)
+- [should return 200 for service type=ExternalName using a port name](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L183)
+- [should return 200 for service type=ExternalName using FQDN with trailing dot](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L216)
+- [should update the external name after a service update](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L247)
+- [should sync ingress on external name service addition/deletion](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L310)
 
-### [[Service] backend status code 503](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_backend.go#L32)
+### [[Service] Nil Service Backend](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_nil_backend.go#L31)
 
-- [should return 503 when backend service does not exist](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_backend.go#L35)
-- [should return 503 when all backend service endpoints are unavailable](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_backend.go#L53)
+- [should return 404 when backend service is nil](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_nil_backend.go#L38)
 
-### [[Service] Type ExternalName](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L37)
+### [[Service] backend status code 503](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_backend.go#L33)
 
-- [works with external name set to incomplete fqdn](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L40)
-- [should return 200 for service type=ExternalName without a port defined](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L73)
-- [should return 200 for service type=ExternalName with a port defined](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L107)
-- [should return status 502 for service type=ExternalName with an invalid host](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L148)
-- [should return 200 for service type=ExternalName using a port name](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L179)
-- [should return 200 for service type=ExternalName using FQDN with trailing dot](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L221)
-- [should update the external name after a service update](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_externalname.go#L252)
+- [should return 503 when backend service does not exist](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_backend.go#L36)
+- [should return 503 when all backend service endpoints are unavailable](https://github.com/kubernetes/ingress-nginx/tree/main/test/e2e/servicebackend/service_backend.go#L54)
diff --git a/hack/generate-e2e-suite-doc.sh b/hack/generate-e2e-suite-doc.sh
index c5545945e..2265bb079 100755
--- a/hack/generate-e2e-suite-doc.sh
+++ b/hack/generate-e2e-suite-doc.sh
@@ -25,9 +25,12 @@ set -o pipefail
 URL="https://github.com/kubernetes/ingress-nginx/tree/main/"
 DIR=$(cd $(dirname "${BASH_SOURCE}")/.. && pwd -P)
 
-echo "
+echo "<!---
+This file is autogenerated!
+Do not try to edit it manually.
+-->
 
-# e2e test suite for [NGINX Ingress Controller]($URL)
+# e2e test suite for [Ingress NGINX Controller]($URL)
 
 "
 
@@ -37,7 +40,7 @@ for FILE in `find $DIR/test/e2e -name "*.go"`;do
     # line number
     DESCRIBE_LINE=$(echo $DESCRIBE | cut -f1 -d ':')
     # clean describe, extracting the string
-    DESCRIBE=$(echo $DESCRIBE | sed -En 's/.*"(.*)".*/\1/p')
+    DESCRIBE=$(echo $DESCRIBE | sed -En 's/.*("|`)(.*)("|`).*/\2/p')
 
     FILE_URL=$(echo $FILE | sed "s|${DIR}/|${URL}|g")
     echo "
@@ -48,7 +51,7 @@ for FILE in `find $DIR/test/e2e -name "*.go"`;do
     ITS=$(cat $FILE | grep -n -oP 'It\(.*')
     while IFS= read -r line; do
         IT_LINE=$(echo $line | cut -f1 -d ':')
-        IT=$(echo $line | sed -En 's/.*"(.*)".*/\1/p')
+        IT=$(echo $line | sed -En 's/.*("|`)(.*)("|`).*/\2/p')
         echo "- [$IT]($FILE_URL#L$IT_LINE)"
     done <<< "$ITS"
 done
diff --git a/images/test-runner/Makefile b/images/test-runner/Makefile
index e6e047e6c..8ebffc5bb 100644
--- a/images/test-runner/Makefile
+++ b/images/test-runner/Makefile
@@ -39,7 +39,7 @@ build: ensure-buildx
 		--progress=$(PROGRESS) \
 		--pull \
 		--build-arg BASE_IMAGE=$(NGINX_BASE_IMAGE) \
-		--build-arg GOLANG_VERSION=1.17.6 \
+		--build-arg GOLANG_VERSION=1.18.2 \
 		--build-arg ETCD_VERSION=3.4.3-0 \
 		--build-arg K8S_RELEASE=v1.21.3 \
 		--build-arg RESTY_CLI_VERSION=0.27 \
diff --git a/internal/ingress/annotations/authreq/main_test.go b/internal/ingress/annotations/authreq/main_test.go
index eac1870e5..e1325235b 100644
--- a/internal/ingress/annotations/authreq/main_test.go
+++ b/internal/ingress/annotations/authreq/main_test.go
@@ -71,6 +71,13 @@ func buildIngress() *networking.Ingress {
 	}
 }
 
+func boolToString(v bool) string {
+	if v {
+		return "true"
+	}
+	return "false"
+}
+
 func TestAnnotations(t *testing.T) {
 	ing := buildIngress()
 
@@ -86,19 +93,20 @@ func TestAnnotations(t *testing.T) {
 		requestRedirect        string
 		authSnippet            string
 		authCacheKey           string
+		authAlwaysSetCookie    bool
 		expErr                 bool
 	}{
-		{"empty", "", "", "", "", "", "", "", true},
-		{"no scheme", "bar", "bar", "", "", "", "", "", true},
-		{"invalid host", "http://", "http://", "", "", "", "", "", true},
-		{"invalid host (multiple dots)", "http://foo..bar.com", "http://foo..bar.com", "", "", "", "", "", true},
-		{"valid URL", "http://bar.foo.com/external-auth", "http://bar.foo.com/external-auth", "", "", "", "", "", false},
-		{"valid URL - send body", "http://foo.com/external-auth", "http://foo.com/external-auth", "", "POST", "", "", "", false},
-		{"valid URL - send body", "http://foo.com/external-auth", "http://foo.com/external-auth", "", "GET", "", "", "", false},
-		{"valid URL - request redirect", "http://foo.com/external-auth", "http://foo.com/external-auth", "", "GET", "http://foo.com/redirect-me", "", "", false},
-		{"auth snippet", "http://foo.com/external-auth", "http://foo.com/external-auth", "", "", "", "proxy_set_header My-Custom-Header 42;", "", false},
-		{"auth cache ", "http://foo.com/external-auth", "http://foo.com/external-auth", "", "", "", "", "$foo$bar", false},
-		{"redirect param", "http://bar.foo.com/external-auth", "http://bar.foo.com/external-auth", "origUrl", "", "", "", "", false},
+		{"empty", "", "", "", "", "", "", "", false, true},
+		{"no scheme", "bar", "bar", "", "", "", "", "", false, true},
+		{"invalid host", "http://", "http://", "", "", "", "", "", false, true},
+		{"invalid host (multiple dots)", "http://foo..bar.com", "http://foo..bar.com", "", "", "", "", "", false, true},
+		{"valid URL", "http://bar.foo.com/external-auth", "http://bar.foo.com/external-auth", "", "", "", "", "", false, false},
+		{"valid URL - send body", "http://foo.com/external-auth", "http://foo.com/external-auth", "", "POST", "", "", "", false, false},
+		{"valid URL - send body", "http://foo.com/external-auth", "http://foo.com/external-auth", "", "GET", "", "", "", false, false},
+		{"valid URL - request redirect", "http://foo.com/external-auth", "http://foo.com/external-auth", "", "GET", "http://foo.com/redirect-me", "", "", false, false},
+		{"auth snippet", "http://foo.com/external-auth", "http://foo.com/external-auth", "", "", "", "proxy_set_header My-Custom-Header 42;", "", false, false},
+		{"auth cache ", "http://foo.com/external-auth", "http://foo.com/external-auth", "", "", "", "", "$foo$bar", false, false},
+		{"redirect param", "http://bar.foo.com/external-auth", "http://bar.foo.com/external-auth", "origUrl", "", "", "", "", true, false},
 	}
 
 	for _, test := range tests {
@@ -109,6 +117,7 @@ func TestAnnotations(t *testing.T) {
 		data[parser.GetAnnotationWithPrefix("auth-request-redirect")] = test.requestRedirect
 		data[parser.GetAnnotationWithPrefix("auth-snippet")] = test.authSnippet
 		data[parser.GetAnnotationWithPrefix("auth-cache-key")] = test.authCacheKey
+		data[parser.GetAnnotationWithPrefix("auth-always-set-cookie")] = boolToString(test.authAlwaysSetCookie)
 
 		i, err := NewParser(&resolver.Mock{}).Parse(ing)
 		if test.expErr {
@@ -146,6 +155,10 @@ func TestAnnotations(t *testing.T) {
 		if u.AuthCacheKey != test.authCacheKey {
 			t.Errorf("%v: expected \"%v\" but \"%v\" was returned", test.title, test.authCacheKey, u.AuthCacheKey)
 		}
+
+		if u.AlwaysSetCookie != test.authAlwaysSetCookie {
+			t.Errorf("%v: expected \"%v\" but \"%v\" was returned", test.title, test.authAlwaysSetCookie, u.AlwaysSetCookie)
+		}
 	}
 }
 
diff --git a/internal/ingress/controller/template/configmap_test.go b/internal/ingress/controller/template/configmap_test.go
index 1f6ce2fe6..b30e836ee 100644
--- a/internal/ingress/controller/template/configmap_test.go
+++ b/internal/ingress/controller/template/configmap_test.go
@@ -229,6 +229,34 @@ func TestGlobalExternalAuthSigninParsing(t *testing.T) {
 	}
 }
 
+func TestGlobalExternalAlwaysSetCookie(t *testing.T) {
+	testCases := map[string]struct {
+		alwaysSetCookie string
+		result          bool
+	}{
+		"true": {
+			alwaysSetCookie: "true",
+			result:          true,
+		},
+		"false": {
+			alwaysSetCookie: "false",
+		},
+		"set empty": {
+			alwaysSetCookie: "",
+		},
+		"error": {
+			alwaysSetCookie: "error string",
+		},
+	}
+
+	for n, tc := range testCases {
+		cfg := ReadConfig(map[string]string{"global-auth-always-set-cookie": tc.alwaysSetCookie})
+		if cfg.GlobalExternalAuth.AlwaysSetCookie != tc.result {
+			t.Errorf("Testing %v. Expected \"%v\" but \"%v\" was returned", n, tc.result, cfg.GlobalExternalAuth.AlwaysSetCookie)
+		}
+	}
+}
+
 func TestGlobalExternalAuthSigninRedirectParamParsing(t *testing.T) {
 	testCases := map[string]struct {
 		param  string
diff --git a/internal/ingress/metric/collectors/socket_test.go b/internal/ingress/metric/collectors/socket_test.go
index 8542d273c..11af7c06d 100644
--- a/internal/ingress/metric/collectors/socket_test.go
+++ b/internal/ingress/metric/collectors/socket_test.go
@@ -80,12 +80,13 @@ func TestCollector(t *testing.T) {
 	}
 
 	cases := []struct {
-		name            string
-		data            []string
-		metrics         []string
-		wantBefore      string
-		removeIngresses []string
-		wantAfter       string
+		name             string
+		data             []string
+		metrics          []string
+		useStatusClasses bool
+		wantBefore       string
+		removeIngresses  []string
+		wantAfter        string
 	}{
 		{
 			name: "invalid metric object should not increase prometheus metrics",
@@ -371,13 +372,55 @@ func TestCollector(t *testing.T) {
 			wantAfter: `
 			`,
 		},
+		{
+			name: "valid metric object with status classes should update prometheus metrics",
+			data: []string{`[{
+				"host":"testshop.com",
+				"status":"200",
+				"bytesSent":150.0,
+				"method":"GET",
+				"path":"/admin",
+				"requestLength":300.0,
+				"requestTime":60.0,
+				"upstreamName":"test-upstream",
+				"upstreamIP":"1.1.1.1:8080",
+				"upstreamResponseTime":200,
+				"upstreamStatus":"220",
+				"namespace":"test-app-production",
+				"ingress":"web-yml",
+				"service":"test-app",
+				"canary":""
+			}]`},
+			metrics:          []string{"nginx_ingress_controller_response_duration_seconds"},
+			useStatusClasses: true,
+			wantBefore: `
+				# HELP nginx_ingress_controller_response_duration_seconds The time spent on receiving the response from the upstream server
+				# TYPE nginx_ingress_controller_response_duration_seconds histogram
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="0.005"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="0.01"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="0.025"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="0.05"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="0.1"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="0.25"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="0.5"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="1"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="2.5"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="5"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="10"} 0
+				nginx_ingress_controller_response_duration_seconds_bucket{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx",le="+Inf"} 1
+				nginx_ingress_controller_response_duration_seconds_sum{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx"} 200
+				nginx_ingress_controller_response_duration_seconds_count{canary="",controller_class="ingress",controller_namespace="default",controller_pod="pod",host="testshop.com",ingress="web-yml",method="GET",namespace="test-app-production",path="/admin",service="test-app",status="2xx"} 1
+			`,
+			wantAfter: `
+			`,
+		},
 	}
 
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {
 			registry := prometheus.NewPedanticRegistry()
 
-			sc, err := NewSocketCollector("pod", "default", "ingress", true, buckets)
+			sc, err := NewSocketCollector("pod", "default", "ingress", true, c.useStatusClasses, buckets)
 			if err != nil {
 				t.Errorf("%v: unexpected error creating new SocketCollector: %v", c.name, err)
 			}
diff --git a/test/e2e/annotations/auth.go b/test/e2e/annotations/auth.go
index 0ea97d3ef..d0ca5cf12 100644
--- a/test/e2e/annotations/auth.go
+++ b/test/e2e/annotations/auth.go
@@ -313,9 +313,12 @@ var _ = framework.DescribeAnnotation("auth-*", func() {
 			})
 	})
 
-	ginkgo.It("retains cookie set by external authentication server", func() {
+	ginkgo.Context("cookie set by external authentication server", func() {
 		host := "auth-check-cookies"
 
+		var annotations map[string]string
+		var ing1, ing2 *networking.Ingress
+
 		cfg := `#
 events {
 	worker_connections  1024;
@@ -342,40 +345,81 @@ http {
 		location / {
 			return 200;
 		}
+
+		location /error {
+			return 503;
+		}
 	}
 }
 `
+		ginkgo.BeforeEach(func() {
+			f.NGINXWithConfigDeployment("http-cookie-with-error", cfg)
 
-		f.NGINXWithConfigDeployment(framework.HTTPBinService, cfg)
+			e, err := f.KubeClientSet.CoreV1().Endpoints(f.Namespace).Get(context.TODO(), "http-cookie-with-error", metav1.GetOptions{})
+			assert.Nil(ginkgo.GinkgoT(), err)
 
-		e, err := f.KubeClientSet.CoreV1().Endpoints(f.Namespace).Get(context.TODO(), framework.HTTPBinService, metav1.GetOptions{})
-		assert.Nil(ginkgo.GinkgoT(), err)
+			assert.GreaterOrEqual(ginkgo.GinkgoT(), len(e.Subsets), 1, "expected at least one endpoint")
+			assert.GreaterOrEqual(ginkgo.GinkgoT(), len(e.Subsets[0].Addresses), 1, "expected at least one address ready in the endpoint")
 
-		assert.GreaterOrEqual(ginkgo.GinkgoT(), len(e.Subsets), 1, "expected at least one endpoint")
-		assert.GreaterOrEqual(ginkgo.GinkgoT(), len(e.Subsets[0].Addresses), 1, "expected at least one address ready in the endpoint")
+			httpbinIP := e.Subsets[0].Addresses[0].IP
 
-		httpbinIP := e.Subsets[0].Addresses[0].IP
+			annotations = map[string]string{
+				"nginx.ingress.kubernetes.io/auth-url":    fmt.Sprintf("http://%s/cookies/set/alma/armud", httpbinIP),
+				"nginx.ingress.kubernetes.io/auth-signin": "http://$host/auth/start",
+			}
 
-		annotations := map[string]string{
-			"nginx.ingress.kubernetes.io/auth-url":    fmt.Sprintf("http://%s/cookies/set/alma/armud", httpbinIP),
-			"nginx.ingress.kubernetes.io/auth-signin": "http://$host/auth/start",
-		}
+			ing1 = framework.NewSingleIngress(host, "/", host, f.Namespace, "http-cookie-with-error", 80, annotations)
+			f.EnsureIngress(ing1)
 
-		ing := framework.NewSingleIngress(host, "/", host, f.Namespace, framework.EchoService, 80, annotations)
-		f.EnsureIngress(ing)
+			ing2 = framework.NewSingleIngress(host+"-error", "/error", host, f.Namespace, "http-cookie-with-error", 80, annotations)
+			f.EnsureIngress(ing2)
+
+			f.WaitForNginxServer(host, func(server string) bool {
+				return strings.Contains(server, "server_name "+host)
+			})
 
-		f.WaitForNginxServer(host, func(server string) bool {
-			return strings.Contains(server, "server_name auth")
 		})
 
-		f.HTTPTestClient().
-			GET("/").
-			WithHeader("Host", host).
-			WithQuery("a", "b").
-			WithQuery("c", "d").
-			Expect().
-			Status(http.StatusOK).
-			Header("Set-Cookie").Contains("alma=armud")
+		ginkgo.It("user retains cookie by default", func() {
+			f.HTTPTestClient().
+				GET("/").
+				WithHeader("Host", host).
+				WithQuery("a", "b").
+				WithQuery("c", "d").
+				Expect().
+				Status(http.StatusOK).
+				Header("Set-Cookie").Contains("alma=armud")
+		})
+
+		ginkgo.It("user does not retain cookie if upstream returns error status code", func() {
+			f.HTTPTestClient().
+				GET("/error").
+				WithHeader("Host", host).
+				WithQuery("a", "b").
+				WithQuery("c", "d").
+				Expect().
+				Status(http.StatusServiceUnavailable).
+				Header("Set-Cookie").Contains("")
+		})
+
+		ginkgo.It("user with annotated ingress retains cookie if upstream returns error status code", func() {
+			annotations["nginx.ingress.kubernetes.io/auth-always-set-cookie"] = "true"
+			f.UpdateIngress(ing1)
+			f.UpdateIngress(ing2)
+
+			f.WaitForNginxServer(host, func(server string) bool {
+				return strings.Contains(server, "server_name "+host)
+			})
+
+			f.HTTPTestClient().
+				GET("/error").
+				WithHeader("Host", host).
+				WithQuery("a", "b").
+				WithQuery("c", "d").
+				Expect().
+				Status(http.StatusServiceUnavailable).
+				Header("Set-Cookie").Contains("alma=armud")
+		})
 	})
 
 	ginkgo.Context("when external authentication is configured", func() {
diff --git a/test/e2e/settings/global_external_auth.go b/test/e2e/settings/global_external_auth.go
index 1e5bf4301..d514080a0 100644
--- a/test/e2e/settings/global_external_auth.go
+++ b/test/e2e/settings/global_external_auth.go
@@ -17,6 +17,7 @@ limitations under the License.
 package settings
 
 import (
+	"context"
 	"fmt"
 	"net/http"
 	"regexp"
@@ -24,6 +25,7 @@ import (
 
 	"github.com/onsi/ginkgo"
 	"github.com/stretchr/testify/assert"
+	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
 	networking "k8s.io/api/networking/v1"
 	"k8s.io/ingress-nginx/test/e2e/framework"
@@ -150,7 +152,7 @@ var _ = framework.DescribeSetting("[Security] global-auth-url", func() {
 				Status(http.StatusOK)
 		})
 
-		ginkgo.It("should still return status code 200 after auth backend is deleted using cache ", func() {
+		ginkgo.It("should still return status code 200 after auth backend is deleted using cache", func() {
 
 			globalExternalAuthCacheKeySetting := "global-auth-cache-key"
 			globalExternalAuthCacheKey := "foo"
@@ -259,4 +261,101 @@ var _ = framework.DescribeSetting("[Security] global-auth-url", func() {
 
 	})
 
+	ginkgo.Context("cookie set by external authentication server", func() {
+		host := "global-external-auth-check-cookies"
+		var ing1, ing2 *networking.Ingress
+
+		cfg := `#
+events {
+	worker_connections  1024;
+	multi_accept on;
+}
+
+http {
+	default_type 'text/plain';
+	client_max_body_size 0;
+
+	server {
+		access_log on;
+		access_log /dev/stdout;
+
+		listen 80;
+
+		location ~ ^/cookies/set/(?<key>.*)/(?<value>.*) {
+			content_by_lua_block {
+				ngx.header['Set-Cookie'] = {ngx.var.key.."="..ngx.var.value}
+				ngx.say("OK")
+			}
+		}
+
+		location / {
+			return 200;
+		}
+
+		location /error {
+			return 503;
+		}
+	}
+}
+`
+		ginkgo.BeforeEach(func() {
+			f.NGINXWithConfigDeployment("http-cookie-with-error", cfg)
+
+			e, err := f.KubeClientSet.CoreV1().Endpoints(f.Namespace).Get(context.TODO(), "http-cookie-with-error", metav1.GetOptions{})
+			assert.Nil(ginkgo.GinkgoT(), err)
+
+			assert.GreaterOrEqual(ginkgo.GinkgoT(), len(e.Subsets), 1, "expected at least one endpoint")
+			assert.GreaterOrEqual(ginkgo.GinkgoT(), len(e.Subsets[0].Addresses), 1, "expected at least one address ready in the endpoint")
+
+			httpbinIP := e.Subsets[0].Addresses[0].IP
+
+			f.UpdateNginxConfigMapData(globalExternalAuthURLSetting, fmt.Sprintf("http://%s/cookies/set/alma/armud", httpbinIP))
+
+			ing1 = framework.NewSingleIngress(host, "/", host, f.Namespace, "http-cookie-with-error", 80, nil)
+			f.EnsureIngress(ing1)
+
+			ing2 = framework.NewSingleIngress(host+"-error", "/error", host, f.Namespace, "http-cookie-with-error", 80, nil)
+			f.EnsureIngress(ing2)
+
+			f.WaitForNginxServer(host, func(server string) bool {
+				return strings.Contains(server, "server_name "+host)
+			})
+
+		})
+
+		ginkgo.It("user retains cookie by default", func() {
+			f.HTTPTestClient().
+				GET("/").
+				WithHeader("Host", host).
+				WithQuery("a", "b").
+				WithQuery("c", "d").
+				Expect().
+				Status(http.StatusOK).
+				Header("Set-Cookie").Contains("alma=armud")
+		})
+
+		ginkgo.It("user does not retain cookie if upstream returns error status code", func() {
+			f.HTTPTestClient().
+				GET("/error").
+				WithHeader("Host", host).
+				WithQuery("a", "b").
+				WithQuery("c", "d").
+				Expect().
+				Status(http.StatusServiceUnavailable).
+				Header("Set-Cookie").Contains("")
+		})
+
+		ginkgo.It("user with global-auth-always-set-cookie key in configmap retains cookie if upstream returns error status code", func() {
+			f.UpdateNginxConfigMapData("global-auth-always-set-cookie", "true")
+
+			f.HTTPTestClient().
+				GET("/error").
+				WithHeader("Host", host).
+				WithQuery("a", "b").
+				WithQuery("c", "d").
+				Expect().
+				Status(http.StatusServiceUnavailable).
+				Header("Set-Cookie").Contains("alma=armud")
+		})
+	})
 })