Repository: kubernetes/ingress-nginx
Pull request #6955 (2021-03-13 14:35:09)

perf: json encoding share to eatch request
<!--- Provide a general summary of your changes in the Title above --->
<!--- Please don't @-mention people in PR or commit messages (do so in an additional comment). --->

## What this PR does / why we need it:
<!--- Why is this change required? What problem does it solve? -->
<!--- If it fixes an open issue, please link to the issue here. -->

## Types of changes
<!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: -->
- [ ] Bug fix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing functionality to change)

## Which issue/s this PR fixes
<!--
(optional, in `fixes #<issue number>` format, will close that issue when PR gets merged):
fixes #
-->
https://github.com/kubernetes/ingress-nginx/issues/6953

## How Has This Been Tested?
<!--- Please describe in detail how you tested your changes. -->
<!--- Include details of your testing environment, and the tests you ran to -->
<!--- see how your change affects other areas of the code, etc. -->

## Checklist:
<!--- Go over all the following points, and put an `x` in all the boxes that apply. -->
<!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! -->
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [ ] I've read the [CONTRIBUTION](https://github.com/kubernetes/ingress-nginx/blob/master/CONTRIBUTING.md) guide
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.


k8s-ci-robot: Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: **Please follow instructions at <https://git.k8s.io/community/CLA.md#the-contributor-license-agreement> to sign the CLA.**

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify.  Thanks.

---

- If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address.  Check your existing CLA data and verify that your [email is set on your git commits](https://help.github.com/articles/setting-your-email-in-git/).
- If you signed the CLA as a corporation, please sign in with your organization's credentials at <https://identity.linuxfoundation.org/projects/cncf> to be authorized.
- If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: <https://support.linuxfoundation.org/>
- Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

<!-- need_sender_cla -->

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
	
k8s-ci-robot: Welcome @zhaogaolong! <br><br>It looks like this is your first PR to <a href='https://github.com/kubernetes/ingress-nginx'>kubernetes/ingress-nginx</a> 🎉. Please refer to our [pull request process documentation](https://git.k8s.io/community/contributors/guide/pull-requests.md) to help your PR have a smooth ride to approval. <br><br>You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. [Here is the bot commands documentation](https://go.k8s.io/bot-commands). <br><br>You can also check if kubernetes/ingress-nginx has [its own contribution guidelines](https://github.com/kubernetes/ingress-nginx/tree/master/CONTRIBUTING.md). <br><br>You may want to refer to our [testing guide](https://git.k8s.io/community/contributors/devel/sig-testing/testing.md) if you run into trouble with your tests not passing. <br><br>If you are having difficulty getting your pull request seen, please follow the [recommended escalation practices](https://github.com/kubernetes/community/blob/master/contributors/guide/pull-requests.md#why-is-my-pull-request-not-getting-reviewed). Also, for tips and tricks in the contribution process you may want to read the [Kubernetes contributor cheat sheet](https://git.k8s.io/community/contributors/guide/contributor-cheatsheet/README.md). We want to make sure your contribution gets all the attention it needs! <br><br>Thank you, and welcome to Kubernetes. :smiley:
k8s-ci-robot: Hi @zhaogaolong. Thanks for your PR.

I'm waiting for a [kubernetes](https://github.com/orgs/kubernetes/people) member to verify that this patch is reasonable to test. If it is, they should reply with `/ok-to-test` on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should [join the org](https://git.k8s.io/community/community-membership.md#member) to skip this step.

Once the patch is verified, the new status will be reflected by the `ok-to-test` label.

I understand the commands that are listed [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fingress-nginx).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>

k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/ingress-nginx/pull/6955#" title="Author self-approved">zhaogaolong</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **bowei** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @bowei` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fingress-nginx).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes/ingress-nginx/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["bowei"]} -->
rikatz: @moonming thoughts on this?
k8s-ci-robot: [APPROVALNOTIFIER] This PR is **NOT APPROVED**

This pull-request has been approved by: *<a href="https://github.com/kubernetes/ingress-nginx/pull/6955#pullrequestreview-648793193" title="Approved">tokers</a>*, *<a href="https://github.com/kubernetes/ingress-nginx/pull/6955#" title="Author self-approved">zhaogaolong</a>*
To complete the [pull request process](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process), please assign **bowei** after the PR has been reviewed.
You can assign the PR to them by writing `/assign @bowei` in a comment when ready.

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=kubernetes%2Fingress-nginx).

<details open>
Needs approval from an approver in each of these files:

- **[OWNERS](https://github.com/kubernetes/ingress-nginx/blob/master/OWNERS)**

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":["bowei"]} -->
zhaogaolong: > @moonming thoughts on this?

When sending metrics data to promethues sock, it will affect the current request delay, because cjson.encoding must be done once before sending, which will affect P95 P99.

If the work of cjson.encoding is allocated to each request, the delay of p95 p99 is greatly reduced
tokers: > > @moonming thoughts on this?
> 
> When sending metrics data to promethues sock, it will affect the current request delay, because cjson.encoding must be done once before sending, which will affect P95 P99.
> 
> If the work of cjson.encoding is allocated to each request, the delay of p95 p99 is greatly reduced
> 
> ## test recode
> test command: `./wrk2 -t2 -c1000 -d180s -L -R30000 http://ingress-nginx-pod-ip`
> 
> x	p75	P95	P99	Remarks
> 1	2.19ms	3.313ms	4.70ms	No lua config
> 2	1.64ms	17.727ms	50.72ms	only config monitor.call()
> 3	1.39ms	5.211ms	9.44ms	used this pr fixed
> @moonming

Good optimization.
k8s-ci-robot: @zhaogaolong: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-ingress-nginx-lualint | cc1514f0548a9d7f9802658829d04ad3fae32e0f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/ingress-nginx/6955/pull-ingress-nginx-lualint/1390508204487610368) | `/test pull-ingress-nginx-lualint`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes&repo=ingress-nginx&pr=6955). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Azhaogaolong). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes/ingress-nginx/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @zhaogaolong: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-ingress-nginx-lualint | cc1514f0548a9d7f9802658829d04ad3fae32e0f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/ingress-nginx/6955/pull-ingress-nginx-lualint/1390508204487610368) | `/test pull-ingress-nginx-lualint`
pull-ingress-nginx-test-lua | cc1514f0548a9d7f9802658829d04ad3fae32e0f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/ingress-nginx/6955/pull-ingress-nginx-test-lua/1390508204512776192) | `/test pull-ingress-nginx-test-lua`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes&repo=ingress-nginx&pr=6955). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Azhaogaolong). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes/ingress-nginx/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
tao12345666333: ```
Checking rootfs/etc/nginx/lua/monitor.lua         1 warning
    rootfs/etc/nginx/lua/monitor.lua:72:9: (W311) value assigned to variable item is unused
Total: 1 warning / 0 errors in 24 files
```
k8s-ci-robot: @zhaogaolong: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-ingress-nginx-lualint | cc1514f0548a9d7f9802658829d04ad3fae32e0f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/ingress-nginx/6955/pull-ingress-nginx-lualint/1390508204487610368) | `/test pull-ingress-nginx-lualint`
pull-ingress-nginx-test-lua | 773d2d6da7296703b4fb7e8654843e48b28b4043 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/ingress-nginx/6955/pull-ingress-nginx-test-lua/1390991863887958016) | `/test pull-ingress-nginx-test-lua`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes&repo=ingress-nginx&pr=6955). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Azhaogaolong). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes/ingress-nginx/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @zhaogaolong: The following tests **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-ingress-nginx-test-lua | 773d2d6da7296703b4fb7e8654843e48b28b4043 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/ingress-nginx/6955/pull-ingress-nginx-test-lua/1390991863887958016) | `/test pull-ingress-nginx-test-lua`
pull-ingress-nginx-lualint | 28986f83cb92697b1157a592cceb306d809e671f | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/ingress-nginx/6955/pull-ingress-nginx-lualint/1391228028951990272) | `/test pull-ingress-nginx-lualint`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes&repo=ingress-nginx&pr=6955). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Azhaogaolong). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes/ingress-nginx/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
tao12345666333: @zhaogaolong  Have you used the latest code for performance testing? What's the result?
zhaogaolong: > used

I'm sorry, I'm sorry I can't do the latest performance test because I left my original company. Are you worried？
tao12345666333: The current code is quite different from your initial code, I would like to know the performance of the latest code.
k8s-ci-robot: @zhaogaolong: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-ingress-nginx-lualint | a7888f09f540eedbaa4055c851a66c352ca81674 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/ingress-nginx/6955/pull-ingress-nginx-lualint/1395745544881573888) | `/test pull-ingress-nginx-lualint`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes&repo=ingress-nginx&pr=6955). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Azhaogaolong). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes/ingress-nginx/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
k8s-ci-robot: @zhaogaolong: The following test **failed**, say `/retest` to rerun all failed tests:

Test name | Commit | Details | Rerun command
--- | --- | --- | ---
pull-ingress-nginx-lualint | 4d2ec5954c954c40741cf2feb4061caf17216bf1 | [link](https://prow.k8s.io/view/gs/kubernetes-jenkins/pr-logs/pull/ingress-nginx/6955/pull-ingress-nginx-lualint/1395769047957639168) | `/test pull-ingress-nginx-lualint`

[Full PR test history](https://prow.k8s.io/pr-history?org=kubernetes&repo=ingress-nginx&pr=6955). [Your PR dashboard](https://prow.k8s.io/pr?query=is%3Apr%20state%3Aopen%20author%3Azhaogaolong). Please help us cut down on flakes by [linking to](https://git.k8s.io/community/contributors/devel/sig-testing/flaky-tests.md#filing-issues-for-flaky-tests) an [open issue](https://github.com/kubernetes/ingress-nginx/issues?q=is:issue+is:open) when you hit one in your PR.

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
zhaogaolong: > The current code is quite different from your initial code, I would like to know the performance of the latest code.

## Performance test report 

test command：`./wrk2 -t10 -c1000 -d180s -L -R30000 http://ingress-nginx-ip`

K8s pod resouece: 8C 12Gi  (Guaranteed)

| version                   | p75    | P95     |   P99   | P99.9    | Times |
| ------------------------- | ------ | ------- | :-----: | -------- | ----- |
| Ingress-nginx      | 1.37ms | 2.143ms | 19.36ms | 248.06ms | 1     |
| Ingress-nginx       | 1.37ms | 2.141ms | 19.61ms | 347.90ms | 2     |
| Ingress-nginx      | 1.43ms | 2.533ms | 19.30ms | 349.95ms | 3     |
| Ingress-nginx-patch | 1.35ms | 1.852ms | 3.41ms  | 7.12ms   | 1     |
| Ingress-nginx-patch | 1.34ms | 1.843ms | 3.46ms  | 8.93ms   | 2     |
| Ingress-nginx-patch | 1.40ms | 2.055ms | 3.73ms  | 7.47ms   | 3     |


Patch:
diff --git a/charts/ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml b/charts/ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
index d9ca4607c..e715c15e5 100644
--- a/charts/ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
+++ b/charts/ingress-nginx/templates/admission-webhooks/job-patch/job-createSecret.yaml
@@ -33,7 +33,7 @@ spec:
       containers:
         - name: create
           {{- with .Values.controller.admissionWebhooks.patch.image }}
-          image: "{{.repository}}:{{ .tag }}{{- if (.digest) -}} @{{.digest}} {{- end -}}"
+          image: "{{- if .repository -}}{{ .repository }}{{ else }}{{ .registry }}/{{ .image }}{{- end -}}:{{ .tag }}{{- if (.digest) -}} @{{.digest}} {{- end -}}"
           {{- end }}
           imagePullPolicy: {{ .Values.controller.admissionWebhooks.patch.image.pullPolicy }}
           args:
diff --git a/charts/ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml b/charts/ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
index d297854cb..b8585e2cf 100644
--- a/charts/ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
+++ b/charts/ingress-nginx/templates/admission-webhooks/job-patch/job-patchWebhook.yaml
@@ -33,7 +33,7 @@ spec:
       containers:
         - name: patch
           {{- with .Values.controller.admissionWebhooks.patch.image }}
-          image: "{{.repository}}:{{ .tag }}{{- if (.digest) -}} @{{.digest}} {{- end -}}"
+          image: "{{- if .repository -}}{{ .repository }}{{ else }}{{ .registry }}/{{ .image }}{{- end -}}:{{ .tag }}{{- if (.digest) -}} @{{.digest}} {{- end -}}"
           {{- end }}
           imagePullPolicy: {{ .Values.controller.admissionWebhooks.patch.image.pullPolicy }}
           args:
diff --git a/charts/ingress-nginx/templates/controller-daemonset.yaml b/charts/ingress-nginx/templates/controller-daemonset.yaml
index 0a8ed3214..991dc4f36 100644
--- a/charts/ingress-nginx/templates/controller-daemonset.yaml
+++ b/charts/ingress-nginx/templates/controller-daemonset.yaml
@@ -64,7 +64,7 @@ spec:
       containers:
         - name: {{ .Values.controller.containerName }}
           {{- with .Values.controller.image }}
-          image: "{{.repository}}:{{ .tag }}{{- if (.digest) -}} @{{.digest}} {{- end -}}"
+          image: "{{- if .repository -}}{{ .repository }}{{ else }}{{ .registry }}/{{ .image }}{{- end -}}:{{ .tag }}{{- if (.digest) -}} @{{.digest}} {{- end -}}"
           {{- end }}
           imagePullPolicy: {{ .Values.controller.image.pullPolicy }}
         {{- if .Values.controller.lifecycle }}
diff --git a/charts/ingress-nginx/templates/controller-deployment.yaml b/charts/ingress-nginx/templates/controller-deployment.yaml
index d1ed71f9a..fe3b56a73 100644
--- a/charts/ingress-nginx/templates/controller-deployment.yaml
+++ b/charts/ingress-nginx/templates/controller-deployment.yaml
@@ -68,7 +68,7 @@ spec:
       containers:
         - name: {{ .Values.controller.containerName }}
           {{- with .Values.controller.image }}
-          image: "{{.repository}}:{{ .tag }}{{- if (.digest) -}} @{{.digest}} {{- end -}}"
+          image: "{{- if .repository -}}{{ .repository }}{{ else }}{{ .registry }}/{{ .image }}{{- end -}}:{{ .tag }}{{- if (.digest) -}} @{{.digest}} {{- end -}}"
           {{- end }}
           imagePullPolicy: {{ .Values.controller.image.pullPolicy }}
         {{- if .Values.controller.lifecycle }}
diff --git a/charts/ingress-nginx/templates/default-backend-deployment.yaml b/charts/ingress-nginx/templates/default-backend-deployment.yaml
index 07c0df740..ff8a6d9a6 100644
--- a/charts/ingress-nginx/templates/default-backend-deployment.yaml
+++ b/charts/ingress-nginx/templates/default-backend-deployment.yaml
@@ -39,7 +39,7 @@ spec:
       containers:
         - name: {{ template "ingress-nginx.name" . }}-default-backend
           {{- with .Values.defaultBackend.image }}
-          image: "{{.repository}}:{{ .tag }}{{- if (.digest) -}} @{{.digest}} {{- end -}}"
+          image: "{{- if .repository -}}{{ .repository }}{{ else }}{{ .registry }}/{{ .image }}{{- end -}}:{{ .tag }}{{- if (.digest) -}} @{{.digest}} {{- end -}}"
           {{- end }}
           imagePullPolicy: {{ .Values.defaultBackend.image.pullPolicy }}
         {{- if .Values.defaultBackend.extraArgs }}
diff --git a/charts/ingress-nginx/values.yaml b/charts/ingress-nginx/values.yaml
index cd1f90760..0f66c48ef 100644
--- a/charts/ingress-nginx/values.yaml
+++ b/charts/ingress-nginx/values.yaml
@@ -10,7 +10,10 @@
 controller:
   name: controller
   image:
-    repository: k8s.gcr.io/ingress-nginx/controller
+    registry: k8s.gcr.io
+    image: ingress-nginx/controller
+    # for backwards compatibility setting full image url via repository value
+    repository:
     tag: "v0.46.0"
     digest: sha256:52f0058bed0a17ab0fb35628ba97e8d52b5d32299fbc03cc0f6c7b9ff036b61a
     pullPolicy: IfNotPresent
@@ -514,7 +517,10 @@ controller:
     patch:
       enabled: true
       image:
-        repository: docker.io/jettech/kube-webhook-certgen
+        registry: docker.io
+        image: jettech/kube-webhook-certgen
+        # for backwards compatibility setting full image url via repository value
+        repository:
         tag: v1.5.1
         pullPolicy: IfNotPresent
       ## Provide a priority class name to the webhook patching job
@@ -631,7 +637,10 @@ defaultBackend:
 
   name: defaultbackend
   image:
-    repository: k8s.gcr.io/defaultbackend-amd64
+    registry: k8s.gcr.io
+    image: defaultbackend-amd64
+    # for backwards compatibility setting full image url via repository value
+    repository:
     tag: "1.5"
     pullPolicy: IfNotPresent
     # nobody user -> uid 65534
diff --git a/docs/user-guide/nginx-configuration/annotations.md b/docs/user-guide/nginx-configuration/annotations.md
index add5c5595..86a6e28e8 100755
--- a/docs/user-guide/nginx-configuration/annotations.md
+++ b/docs/user-guide/nginx-configuration/annotations.md
@@ -348,10 +348,10 @@ CORS can be controlled with the following annotations:
   - Example: `nginx.ingress.kubernetes.io/cors-expose-headers: "*, X-CustomResponseHeader"`
 
 * `nginx.ingress.kubernetes.io/cors-allow-origin`
-  controls what's the accepted Origin for CORS.
-  This is a single field value, with the following format: `http(s)://origin-site.com` or `http(s)://origin-site.com:port`
+  controls which Origin(s) are allowed for CORS.
+  This is a multi-valued field, separated by ',', each value should obey the following format: `http(s)://origin-site.com` or `http(s)://origin-site.com:port`
   - Default: `*`
-  - Example: `nginx.ingress.kubernetes.io/cors-allow-origin: "https://origin-site.com:4443"`
+  - Example: `nginx.ingress.kubernetes.io/cors-allow-origin: "https://origin-site.com:4443, https://origin-site-2.com"`
 
 * `nginx.ingress.kubernetes.io/cors-allow-credentials`
   controls if credentials can be passed during CORS operations.
diff --git a/internal/ingress/annotations/cors/main.go b/internal/ingress/annotations/cors/main.go
index 2f1a0a37b..716d624fa 100644
--- a/internal/ingress/annotations/cors/main.go
+++ b/internal/ingress/annotations/cors/main.go
@@ -18,6 +18,7 @@ package cors
 
 import (
 	"regexp"
+	"strings"
 
 	networking "k8s.io/api/networking/v1beta1"
 
@@ -113,7 +114,14 @@ func (c cors) Parse(ing *networking.Ingress) (interface{}, error) {
 	}
 
 	config.CorsAllowOrigin, err = parser.GetStringAnnotation("cors-allow-origin", ing)
-	if err != nil || !corsOriginRegex.MatchString(config.CorsAllowOrigin) {
+	if err == nil {
+		for _, origin := range strings.Split(config.CorsAllowOrigin, ",") {
+			if !corsOriginRegex.MatchString(strings.TrimSpace(origin)) {
+				config.CorsAllowOrigin = "*"
+				break
+			}
+		}
+	} else {
 		config.CorsAllowOrigin = "*"
 	}
 
diff --git a/internal/ingress/annotations/rewrite/main.go b/internal/ingress/annotations/rewrite/main.go
index 67d0e42bf..902f00f4c 100644
--- a/internal/ingress/annotations/rewrite/main.go
+++ b/internal/ingress/annotations/rewrite/main.go
@@ -35,6 +35,8 @@ type Config struct {
 	SSLRedirect bool `json:"sslRedirect"`
 	// ForceSSLRedirect indicates if the location section is accessible SSL only
 	ForceSSLRedirect bool `json:"forceSSLRedirect"`
+	// PreserveTrailingSlash indicates if the trailing slash should be kept during a tls redirect
+	PreserveTrailingSlash bool `json:"preserveTrailingSlash"`
 	// AppRoot defines the Application Root that the Controller must redirect if it's in '/' context
 	AppRoot string `json:"appRoot"`
 	// UseRegex indicates whether or not the locations use regex paths
@@ -88,6 +90,10 @@ func (a rewrite) Parse(ing *networking.Ingress) (interface{}, error) {
 	if err != nil {
 		config.SSLRedirect = a.r.GetDefaultBackend().SSLRedirect
 	}
+	config.PreserveTrailingSlash, err = parser.GetBoolAnnotation("preserve-trailing-slash", ing)
+	if err != nil {
+		config.PreserveTrailingSlash = a.r.GetDefaultBackend().PreserveTrailingSlash
+	}
 
 	config.ForceSSLRedirect, err = parser.GetBoolAnnotation("force-ssl-redirect", ing)
 	if err != nil {
diff --git a/internal/ingress/controller/config/config.go b/internal/ingress/controller/config/config.go
index 1da22b21e..f8e79e66e 100644
--- a/internal/ingress/controller/config/config.go
+++ b/internal/ingress/controller/config/config.go
@@ -850,6 +850,7 @@ func NewDefault() Configuration {
 			ProxyRequestBuffering:    "on",
 			ProxyRedirectFrom:        "off",
 			ProxyRedirectTo:          "off",
+			PreserveTrailingSlash:    false,
 			SSLRedirect:              true,
 			CustomHTTPErrors:         []int{},
 			WhitelistSourceRange:     []string{},
diff --git a/internal/ingress/controller/template/template.go b/internal/ingress/controller/template/template.go
index 6cdc1d7b2..98e737a63 100644
--- a/internal/ingress/controller/template/template.go
+++ b/internal/ingress/controller/template/template.go
@@ -342,12 +342,14 @@ func locationConfigForLua(l interface{}, a interface{}) string {
 		force_ssl_redirect = %t,
 		ssl_redirect = %t,
 		force_no_ssl_redirect = %t,
+		preserve_trailing_slash = %t,
 		use_port_in_redirects = %t,
 		global_throttle = { namespace = "%v", limit = %d, window_size = %d, key = %v, ignored_cidrs = %v },
 	}`,
 		location.Rewrite.ForceSSLRedirect,
 		location.Rewrite.SSLRedirect,
 		isLocationInLocationList(l, all.Cfg.NoTLSRedirectLocations),
+		location.Rewrite.PreserveTrailingSlash,
 		location.UsePortInRedirects,
 		location.GlobalRateLimit.Namespace,
 		location.GlobalRateLimit.Limit,
diff --git a/internal/ingress/defaults/main.go b/internal/ingress/defaults/main.go
index f3034d202..03926baa0 100644
--- a/internal/ingress/defaults/main.go
+++ b/internal/ingress/defaults/main.go
@@ -31,6 +31,9 @@ type Backend struct {
 	// By default this is disabled
 	CustomHTTPErrors []int `json:"custom-http-errors"`
 
+	// toggles whether or not to remove trailing slashes during TLS redirects
+	PreserveTrailingSlash bool `json:"preserve-trailing-slash"`
+
 	// http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size
 	// Sets the maximum allowed size of the client request body
 	ProxyBodySize string `json:"proxy-body-size"`
diff --git a/mkdocs.yml b/mkdocs.yml
index 82b00fc86..e15e015dc 100644
--- a/mkdocs.yml
+++ b/mkdocs.yml
@@ -72,7 +72,6 @@ nav:
       - How it works: "how-it-works.md"
       - Troubleshooting: "troubleshooting.md"
       - kubectl plugin: "kubectl-plugin.md"
-      - Development: "development.md"
   - Deployment:
       - Installation Guide: "deploy/index.md"
       - Bare-metal considerations: "deploy/baremetal.md"
diff --git a/rootfs/etc/nginx/lua/lua_ingress.lua b/rootfs/etc/nginx/lua/lua_ingress.lua
index 90042fbd3..49e0f5b05 100644
--- a/rootfs/etc/nginx/lua/lua_ingress.lua
+++ b/rootfs/etc/nginx/lua/lua_ingress.lua
@@ -147,9 +147,11 @@ function _M.rewrite(location_config)
 
   if redirect_to_https(location_config) then
     local request_uri = ngx.var.request_uri
-    -- do not append a trailing slash on redirects
-    if string.sub(request_uri, -1) == "/" then
-      request_uri = string.sub(request_uri, 1, -2)
+    -- do not append a trailing slash on redirects unless enabled by annotations
+    if location_config.preserve_trailing_slash == false then
+      if string.byte(request_uri, -1, -1) == string.byte('/') then
+        request_uri = string.sub(request_uri, 1, -2)
+      end
     end
 
     local uri = string_format("https://%s%s", redirect_host(), request_uri)
diff --git a/rootfs/etc/nginx/lua/monitor.lua b/rootfs/etc/nginx/lua/monitor.lua
index 1c0672e41..3e7d6b34c 100644
--- a/rootfs/etc/nginx/lua/monitor.lua
+++ b/rootfs/etc/nginx/lua/monitor.lua
@@ -7,7 +7,8 @@ local socket = ngx.socket.tcp
 local cjson = require("cjson.safe")
 local new_tab = require "table.new"
 local clear_tab = require "table.clear"
-local clone_tab = require "table.clone"
+local table = table
+local pairs = pairs
 
 
 -- if an Nginx worker processes more than (MAX_BATCH_SIZE/FLUSH_INTERVAL) RPS
@@ -18,6 +19,9 @@ local FLUSH_INTERVAL = 1 -- second
 local metrics_batch = new_tab(MAX_BATCH_SIZE, 0)
 local metrics_count = 0
 
+-- for save json raw metrics table
+local metrics_raw_batch = new_tab(MAX_BATCH_SIZE, 0)
+
 local _M = {}
 
 local function send(payload)
@@ -57,16 +61,22 @@ local function flush(premature)
     return
   end
 
-  local current_metrics_batch = clone_tab(metrics_batch)
-  clear_tab(metrics_batch)
   metrics_count = 0
+  clear_tab(metrics_batch)
 
-  local payload, err = cjson.encode(current_metrics_batch)
-  if not payload then
-    ngx.log(ngx.ERR, "error while encoding metrics: ", err)
-    return
+  local request_metrics = {}
+  table.insert(request_metrics, "[")
+  for i in pairs(metrics_raw_batch) do
+    local item = metrics_raw_batch[i] ..","
+    if i == table.getn(metrics_raw_batch) then
+      item = metrics_raw_batch[i]
+    end
+    table.insert(request_metrics, item)
   end
+  table.insert(request_metrics, "]")
+  local payload = table.concat(request_metrics)
 
+  clear_tab(metrics_raw_batch)
   send(payload)
 end
 
@@ -90,8 +100,16 @@ function _M.call()
     return
   end
 
+  local metrics_obj = metrics()
+  local payload, err = cjson.encode(metrics_obj)
+  if err then
+    ngx.log(ngx.ERR, string.format("error when encoding metrics: %s", tostring(err)))
+    return
+  end
+
   metrics_count = metrics_count + 1
-  metrics_batch[metrics_count] = metrics()
+  metrics_batch[metrics_count] = metrics_obj
+  metrics_raw_batch[metrics_count] = payload
 end
 
 setmetatable(_M, {__index = {


Test patch:
diff --git a/internal/ingress/annotations/cors/main_test.go b/internal/ingress/annotations/cors/main_test.go
index 6f75ce6a7..5715925ed 100644
--- a/internal/ingress/annotations/cors/main_test.go
+++ b/internal/ingress/annotations/cors/main_test.go
@@ -72,7 +72,7 @@ func TestIngressCorsConfigValid(t *testing.T) {
 	data[parser.GetAnnotationWithPrefix("cors-allow-headers")] = "DNT,X-CustomHeader, Keep-Alive,User-Agent"
 	data[parser.GetAnnotationWithPrefix("cors-allow-credentials")] = "false"
 	data[parser.GetAnnotationWithPrefix("cors-allow-methods")] = "GET, PATCH"
-	data[parser.GetAnnotationWithPrefix("cors-allow-origin")] = "https://origin123.test.com:4443"
+	data[parser.GetAnnotationWithPrefix("cors-allow-origin")] = "https://origin123.test.com:4443, https://origin456.test.com"
 	data[parser.GetAnnotationWithPrefix("cors-expose-headers")] = "*, X-CustomResponseHeader"
 	data[parser.GetAnnotationWithPrefix("cors-max-age")] = "600"
 	ing.SetAnnotations(data)
@@ -103,7 +103,7 @@ func TestIngressCorsConfigValid(t *testing.T) {
 		t.Errorf("expected %v but returned %v", data[parser.GetAnnotationWithPrefix("cors-allow-methods")], nginxCors.CorsAllowMethods)
 	}
 
-	if nginxCors.CorsAllowOrigin != "https://origin123.test.com:4443" {
+	if nginxCors.CorsAllowOrigin != "https://origin123.test.com:4443, https://origin456.test.com" {
 		t.Errorf("expected %v but returned %v", data[parser.GetAnnotationWithPrefix("cors-allow-origin")], nginxCors.CorsAllowOrigin)
 	}
 
@@ -126,7 +126,7 @@ func TestIngressCorsConfigInvalid(t *testing.T) {
 	data[parser.GetAnnotationWithPrefix("cors-allow-headers")] = "@alright, #ingress"
 	data[parser.GetAnnotationWithPrefix("cors-allow-credentials")] = "no"
 	data[parser.GetAnnotationWithPrefix("cors-allow-methods")] = "GET, PATCH, $nginx"
-	data[parser.GetAnnotationWithPrefix("cors-allow-origin")] = "origin123.test.com:4443"
+	data[parser.GetAnnotationWithPrefix("cors-allow-origin")] = "https://origin.test.com, origin123.test.com:4443"
 	data[parser.GetAnnotationWithPrefix("cors-expose-headers")] = "@alright, #ingress"
 	data[parser.GetAnnotationWithPrefix("cors-max-age")] = "abcd"
 	ing.SetAnnotations(data)
diff --git a/test/e2e/annotations/cors.go b/test/e2e/annotations/cors.go
index e4cdd32b4..526739539 100644
--- a/test/e2e/annotations/cors.go
+++ b/test/e2e/annotations/cors.go
@@ -121,6 +121,22 @@ var _ = framework.DescribeAnnotation("cors-*", func() {
 			})
 	})
 
+	ginkgo.It("should allow multiple origins for cors", func() {
+		host := "cors.foo.com"
+		annotations := map[string]string{
+			"nginx.ingress.kubernetes.io/enable-cors":       "true",
+			"nginx.ingress.kubernetes.io/cors-allow-origin": "https://origin.cors.com:8080, https://origin2.cors.com",
+		}
+
+		ing := framework.NewSingleIngress(host, "/", host, f.Namespace, framework.EchoService, 80, annotations)
+		f.EnsureIngress(ing)
+
+		f.WaitForNginxServer(host,
+			func(server string) bool {
+				return strings.Contains(server, "more_set_headers 'Access-Control-Allow-Origin: https://origin.cors.com:8080, https://origin2.cors.com';")
+			})
+	})
+
 	ginkgo.It("should allow headers for cors", func() {
 		host := "cors.foo.com"
 		annotations := map[string]string{
diff --git a/test/e2e/annotations/preservetrailingslash.go b/test/e2e/annotations/preservetrailingslash.go
new file mode 100644
index 000000000..f9129c77e
--- /dev/null
+++ b/test/e2e/annotations/preservetrailingslash.go
@@ -0,0 +1,53 @@
+/*
+Copyright 2021 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package annotations
+
+import (
+	"net/http"
+
+	"github.com/onsi/ginkgo"
+
+	"k8s.io/ingress-nginx/test/e2e/framework"
+)
+
+var _ = framework.DescribeAnnotation("preserve-trailing-slash", func() {
+	f := framework.NewDefaultFramework("preservetrailingslash")
+
+	ginkgo.BeforeEach(func() {
+		f.NewEchoDeployment()
+	})
+
+	ginkgo.It("should allow preservation of trailing slashes", func() {
+		host := "forcesslredirect.bar.com"
+		tlsHosts := []string{host}
+
+		annotations := map[string]string{
+			"nginx.ingress.kubernetes.io/ssl-redirect":            "true",
+			"nginx.ingress.kubernetes.io/preserve-trailing-slash": "true",
+		}
+
+		ing := framework.NewSingleIngressWithTLS(host, "/", host, tlsHosts, f.Namespace, framework.EchoService, 80, annotations)
+		f.EnsureIngress(ing)
+
+		f.HTTPTestClient().
+			GET("/").
+			WithHeader("Host", host).
+			Expect().
+			Status(http.StatusPermanentRedirect).
+			Header("Location").Equal("https://forcesslredirect.bar.com/")
+	})
+})