Repository: knative-sandbox/eventing-kafka-broker
Pull request #2452 (2022-08-02 08:32:39)

Align backoff delay to old source
As per title

Signed-off-by: Pierangelo Di Pilato <pierdipi@redhat.com>


knative-prow[bot]: [APPROVALNOTIFIER] This PR is **APPROVED**

This pull-request has been approved by: *<a href="https://github.com/knative-sandbox/eventing-kafka-broker/pull/2452#" title="Author self-approved">pierDipi</a>*

The full list of commands accepted by this bot can be found [here](https://go.k8s.io/bot-commands?repo=knative-sandbox%2Feventing-kafka-broker).

The pull request process is described [here](https://git.k8s.io/community/contributors/guide/owners.md#the-code-review-process)

<details >
Needs approval from an approver in each of these files:

- ~~[OWNERS](https://github.com/knative-sandbox/eventing-kafka-broker/blob/main/OWNERS)~~ [pierDipi]

Approvers can indicate their approval by writing `/approve` in a comment
Approvers can cancel approval by writing `/approve cancel` in a comment
</details>
<!-- META={"approvers":[]} -->
pierDipi: /cc @matzew @aavarghese

/cherry-pick release-1.6
knative-prow-robot: @pierDipi: once the present PR merges, I will cherry-pick it on top of release-1.6 in a new PR and assign it to you.

<details>

In response to [this](https://github.com/knative-sandbox/eventing-kafka-broker/pull/2452#issuecomment-1202184548):

>/cc @matzew @aavarghese
>
>/cherry-pick release-1.6


Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
</details>
knative-prow[bot]: @pierDipi: The following test **failed**, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests:

Test name | Commit | Details | Required | Rerun command
--- | --- | --- | --- | ---
upgrade-tests_eventing-kafka-broker_main | 22dd10ae5f27258c2834078fca904e2a151743f4 | [link](https://prow.knative.dev/view/gs/knative-prow/pr-logs/pull/knative-sandbox_eventing-kafka-broker/2452/upgrade-tests_eventing-kafka-broker_main/1554384681607106560) | true | `/test upgrade-tests_eventing-kafka-broker_main`

[Your PR dashboard](https://prow.knative.dev/pr).

<details>

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md).  If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository. I understand the commands that are listed [here](https://go.k8s.io/bot-commands).
</details>
<!-- test report -->
codecov[bot]: # [Codecov](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox) Report
> Merging [#2452](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox) (22dd10a) into [main](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/commit/89b4bb7ac22a1baa62eab29ed03286de0404ccbe?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox) (89b4bb7) will **increase** coverage by `16.59%`.
> The diff coverage is `n/a`.

```diff
@@              Coverage Diff              @@
##               main    #2452       +/-   ##
=============================================
+ Coverage     64.59%   81.18%   +16.59%     
+ Complexity      709      708        -1     
=============================================
  Files           146       72       -74     
  Lines          9806     2466     -7340     
  Branches        224      224               
=============================================
- Hits           6334     2002     -4332     
+ Misses         3039      338     -2701     
+ Partials        433      126      -307     
```

| Flag | Coverage Δ | |
|---|---|---|
| java-unittests | `81.18% <ø> (-0.13%)` | :arrow_down: |

Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#carryforward-flags-in-the-pull-request-comment) to find out more.

| [Impacted Files](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox) | Coverage Δ | |
|---|---|---|
| [...dispatcher/impl/consumer/BaseConsumerVerticle.java](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#diff-ZGF0YS1wbGFuZS9kaXNwYXRjaGVyL3NyYy9tYWluL2phdmEvZGV2L2tuYXRpdmUvZXZlbnRpbmcva2Fma2EvYnJva2VyL2Rpc3BhdGNoZXIvaW1wbC9jb25zdW1lci9CYXNlQ29uc3VtZXJWZXJ0aWNsZS5qYXZh) | `87.09% <0.00%> (-6.46%)` | :arrow_down: |
| [...broker/dispatcher/impl/consumer/OffsetManager.java](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#diff-ZGF0YS1wbGFuZS9kaXNwYXRjaGVyL3NyYy9tYWluL2phdmEvZGV2L2tuYXRpdmUvZXZlbnRpbmcva2Fma2EvYnJva2VyL2Rpc3BhdGNoZXIvaW1wbC9jb25zdW1lci9PZmZzZXRNYW5hZ2VyLmphdmE=) | `94.62% <0.00%> (-1.08%)` | :arrow_down: |
| [control-plane/pkg/reconciler/source/source.go](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#diff-Y29udHJvbC1wbGFuZS9wa2cvcmVjb25jaWxlci9zb3VyY2Uvc291cmNlLmdv) | | |
| [...security/secrets\_provider\_legacy\_channel\_secret.go](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#diff-Y29udHJvbC1wbGFuZS9wa2cvc2VjdXJpdHkvc2VjcmV0c19wcm92aWRlcl9sZWdhY3lfY2hhbm5lbF9zZWNyZXQuZ28=) | | |
| [control-plane/pkg/kafka/consumer\_group\_lag.go](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#diff-Y29udHJvbC1wbGFuZS9wa2cva2Fma2EvY29uc3VtZXJfZ3JvdXBfbGFnLmdv) | | |
| [...l-plane/pkg/reconciler/consumergroup/controller.go](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#diff-Y29udHJvbC1wbGFuZS9wa2cvcmVjb25jaWxlci9jb25zdW1lcmdyb3VwL2NvbnRyb2xsZXIuZ28=) | | |
| [control-plane/pkg/kafka/broker\_class.go](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#diff-Y29udHJvbC1wbGFuZS9wa2cva2Fma2EvYnJva2VyX2NsYXNzLmdv) | | |
| [control-plane/pkg/contract/extensions.go](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#diff-Y29udHJvbC1wbGFuZS9wa2cvY29udHJhY3QvZXh0ZW5zaW9ucy5nbw==) | | |
| [control-plane/pkg/security/secret.go](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#diff-Y29udHJvbC1wbGFuZS9wa2cvc2VjdXJpdHkvc2VjcmV0Lmdv) | | |
| [...ane/pkg/apis/eventing/v1alpha1/kafka\_sink\_types.go](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox#diff-Y29udHJvbC1wbGFuZS9wa2cvYXBpcy9ldmVudGluZy92MWFscGhhMS9rYWZrYV9zaW5rX3R5cGVzLmdv) | | |
| ... and [66 more](https://codecov.io/gh/knative-sandbox/eventing-kafka-broker/pull/2452/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=knative-sandbox).

pierDipi: /retest-requied

Patch:
diff --git a/control-plane/pkg/reconciler/source/source.go b/control-plane/pkg/reconciler/source/source.go
index 2e3a7b2d3..c3536ce35 100644
--- a/control-plane/pkg/reconciler/source/source.go
+++ b/control-plane/pkg/reconciler/source/source.go
@@ -126,7 +126,7 @@ func (r Reconciler) reconcileConsumerGroup(ctx context.Context, ks *sources.Kafk
 						DeliverySpec: &eventingduck.DeliverySpec{
 							Retry:         pointer.Int32(10),
 							BackoffPolicy: &backoffPolicy,
-							BackoffDelay:  pointer.String("PT10S"),
+							BackoffDelay:  pointer.String("PT0.3S"),
 							Timeout:       pointer.String("PT600S"),
 						},
 						Ordering: DefaultDeliveryOrder,
diff --git a/data-plane/config/broker/100-config-kafka-broker-data-plane.yaml b/data-plane/config/broker/100-config-kafka-broker-data-plane.yaml
index 37ffd90c1..f3cbd9989 100644
--- a/data-plane/config/broker/100-config-kafka-broker-data-plane.yaml
+++ b/data-plane/config/broker/100-config-kafka-broker-data-plane.yaml
@@ -115,7 +115,7 @@ data:
     value.deserializer=io.cloudevents.kafka.CloudEventDeserializer
     fetch.min.bytes=1
     heartbeat.interval.ms=3000
-    max.partition.fetch.bytes=1048576
+    max.partition.fetch.bytes=65536
     session.timeout.ms=10000
     # ssl.key.password=
     # ssl.keystore.location=
@@ -132,7 +132,7 @@ data:
     fetch.max.bytes=52428800
     isolation.level=read_uncommitted
     max.poll.interval.ms=300000
-    max.poll.records=500
+    max.poll.records=50
     partition.assignment.strategy=org.apache.kafka.clients.consumer.StickyAssignor
     receive.buffer.bytes=65536
     request.timeout.ms=2000
diff --git a/data-plane/config/channel/100-config-kafka-channel-data-plane.yaml b/data-plane/config/channel/100-config-kafka-channel-data-plane.yaml
index 112ae2ced..1216f2b20 100644
--- a/data-plane/config/channel/100-config-kafka-channel-data-plane.yaml
+++ b/data-plane/config/channel/100-config-kafka-channel-data-plane.yaml
@@ -115,7 +115,7 @@ data:
     value.deserializer=io.cloudevents.kafka.CloudEventDeserializer
     fetch.min.bytes=1
     heartbeat.interval.ms=3000
-    max.partition.fetch.bytes=1048576
+    max.partition.fetch.bytes=65536
     session.timeout.ms=10000
     # ssl.key.password=
     # ssl.keystore.location=
@@ -131,7 +131,7 @@ data:
     fetch.max.bytes=52428800
     isolation.level=read_uncommitted
     max.poll.interval.ms=300000
-    max.poll.records=500
+    max.poll.records=50
     partition.assignment.strategy=org.apache.kafka.clients.consumer.StickyAssignor
     receive.buffer.bytes=65536
     request.timeout.ms=2000
diff --git a/data-plane/config/source/100-config-kafka-source-data-plane.yaml b/data-plane/config/source/100-config-kafka-source-data-plane.yaml
index 810f75a06..27a4f8ef1 100644
--- a/data-plane/config/source/100-config-kafka-source-data-plane.yaml
+++ b/data-plane/config/source/100-config-kafka-source-data-plane.yaml
@@ -97,7 +97,7 @@ data:
     value.deserializer=io.cloudevents.kafka.CloudEventDeserializer
     fetch.min.bytes=1
     heartbeat.interval.ms=3000
-    max.partition.fetch.bytes=1048576
+    max.partition.fetch.bytes=65536
     session.timeout.ms=10000
     # ssl.key.password=
     # ssl.keystore.location=
@@ -114,7 +114,7 @@ data:
     fetch.max.bytes=52428800
     isolation.level=read_uncommitted
     max.poll.interval.ms=300000
-    max.poll.records=500
+    max.poll.records=50
     partition.assignment.strategy=org.apache.kafka.clients.consumer.StickyAssignor
     receive.buffer.bytes=65536
     request.timeout.ms=2000


Test patch:
diff --git a/.github/workflows/kind-e2e.yaml b/.github/workflows/kind-e2e.yaml
index f94a87295..71fd44d20 100644
--- a/.github/workflows/kind-e2e.yaml
+++ b/.github/workflows/kind-e2e.yaml
@@ -24,12 +24,6 @@ jobs:
         eventing-config:
           - "./third_party/eventing-latest/"
 
-        channel-auth-scenario:
-          - ""
-          - "SSL"
-          - "SASL_SSL"
-          - "SASL_PLAIN"
-
         broker-class:
           - Kafka
 
@@ -56,50 +50,6 @@ jobs:
           - test-suite: ./test/e2e_channel
             extra-test-flags: -channels=messaging.knative.dev/v1beta1:KafkaChannel
 
-        # don't run e2e for tests other than KafkaChannel when channel-auth-scenario is set
-        exclude:
-          - test-suite: ./test/e2e
-            channel-auth-scenario: "SSL"
-          - test-suite: ./test/e2e
-            channel-auth-scenario: "SASL_SSL"
-          - test-suite: ./test/e2e
-            channel-auth-scenario: "SASL_PLAIN"
-
-          - test-suite: ./test/e2e_broker
-            channel-auth-scenario: "SSL"
-          - test-suite: ./test/e2e_broker
-            channel-auth-scenario: "SASL_SSL"
-          - test-suite: ./test/e2e_broker
-            channel-auth-scenario: "SASL_PLAIN"
-
-          - test-suite: ./test/e2e_sink
-            channel-auth-scenario: "SSL"
-          - test-suite: ./test/e2e_sink
-            channel-auth-scenario: "SASL_SSL"
-          - test-suite: ./test/e2e_sink
-            channel-auth-scenario: "SASL_PLAIN"
-
-          - test-suite: ./test/e2e_source
-            channel-auth-scenario: "SSL"
-          - test-suite: ./test/e2e_source
-            channel-auth-scenario: "SASL_SSL"
-          - test-suite: ./test/e2e_source
-            channel-auth-scenario: "SASL_PLAIN"
-
-          - test-suite: ./test/e2e_new
-            channel-auth-scenario: "SSL"
-          - test-suite: ./test/e2e_new
-            channel-auth-scenario: "SASL_SSL"
-          - test-suite: ./test/e2e_new
-            channel-auth-scenario: "SASL_PLAIN"
-
-          - test-suite: ./test/experimental
-            channel-auth-scenario: "SSL"
-          - test-suite: ./test/experimental
-            channel-auth-scenario: "SASL_SSL"
-          - test-suite: ./test/experimental
-            channel-auth-scenario: "SASL_PLAIN"
-
     env:
       GOPATH: ${{ github.workspace }}
       KO_DOCKER_REPO: localhost:5001
@@ -115,7 +65,6 @@ jobs:
       NODE_VERSION: ${{ matrix.k8s-version }}
       NODE_SHA: ${{ matrix.kind-image-sha }}
 
-      EVENTING_KAFKA_BROKER_CHANNEL_AUTH_SCENARIO: ${{ matrix.channel-auth-scenario }}
       BROKER_CLASS: ${{ matrix.broker-class }}
 
     steps:
diff --git a/control-plane/pkg/reconciler/source/source_test.go b/control-plane/pkg/reconciler/source/source_test.go
index 09d333442..2ace88d4a 100644
--- a/control-plane/pkg/reconciler/source/source_test.go
+++ b/control-plane/pkg/reconciler/source/source_test.go
@@ -112,7 +112,7 @@ func TestReconcileKind(t *testing.T) {
 								internals.Ordered,
 								NewConsumerTimeout("PT600S"),
 								NewConsumerRetry(10),
-								NewConsumerBackoffDelay("PT10S"),
+								NewConsumerBackoffDelay("PT0.3S"),
 								NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 							),
 						),
@@ -158,7 +158,7 @@ func TestReconcileKind(t *testing.T) {
 								internals.Ordered,
 								NewConsumerTimeout("PT600S"),
 								NewConsumerRetry(10),
-								NewConsumerBackoffDelay("PT10S"),
+								NewConsumerBackoffDelay("PT0.3S"),
 								NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 							),
 						),
@@ -286,7 +286,7 @@ func TestReconcileKind(t *testing.T) {
 								internals.Ordered,
 								NewConsumerTimeout("PT600S"),
 								NewConsumerRetry(10),
-								NewConsumerBackoffDelay("PT10S"),
+								NewConsumerBackoffDelay("PT0.3S"),
 								NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 							),
 						),
@@ -398,7 +398,7 @@ func TestReconcileKind(t *testing.T) {
 								internals.Ordered,
 								NewConsumerTimeout("PT600S"),
 								NewConsumerRetry(10),
-								NewConsumerBackoffDelay("PT10S"),
+								NewConsumerBackoffDelay("PT0.3S"),
 								NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 							),
 						),
@@ -447,7 +447,7 @@ func TestReconcileKind(t *testing.T) {
 								internals.Ordered,
 								NewConsumerTimeout("PT600S"),
 								NewConsumerRetry(10),
-								NewConsumerBackoffDelay("PT10S"),
+								NewConsumerBackoffDelay("PT0.3S"),
 								NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 							),
 						),
@@ -495,7 +495,7 @@ func TestReconcileKind(t *testing.T) {
 								internals.Ordered,
 								NewConsumerTimeout("PT600S"),
 								NewConsumerRetry(10),
-								NewConsumerBackoffDelay("PT10S"),
+								NewConsumerBackoffDelay("PT0.3S"),
 								NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 							),
 						),
@@ -527,7 +527,7 @@ func TestReconcileKind(t *testing.T) {
 									internals.Ordered,
 									NewConsumerTimeout("PT600S"),
 									NewConsumerRetry(10),
-									NewConsumerBackoffDelay("PT10S"),
+									NewConsumerBackoffDelay("PT0.3S"),
 									NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 								),
 							),
@@ -585,7 +585,7 @@ func TestReconcileKind(t *testing.T) {
 									internals.Ordered,
 									NewConsumerTimeout("PT600S"),
 									NewConsumerRetry(10),
-									NewConsumerBackoffDelay("PT10S"),
+									NewConsumerBackoffDelay("PT0.3S"),
 									NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 								),
 							),
@@ -642,7 +642,7 @@ func TestReconcileKind(t *testing.T) {
 									internals.Ordered,
 									NewConsumerTimeout("PT600S"),
 									NewConsumerRetry(10),
-									NewConsumerBackoffDelay("PT10S"),
+									NewConsumerBackoffDelay("PT0.3S"),
 									NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 								),
 							),
@@ -685,7 +685,7 @@ func TestReconcileKind(t *testing.T) {
 								internals.Ordered,
 								NewConsumerTimeout("PT600S"),
 								NewConsumerRetry(10),
-								NewConsumerBackoffDelay("PT10S"),
+								NewConsumerBackoffDelay("PT0.3S"),
 								NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 							),
 						),
@@ -729,7 +729,7 @@ func TestReconcileKind(t *testing.T) {
 								internals.Ordered,
 								NewConsumerTimeout("PT600S"),
 								NewConsumerRetry(10),
-								NewConsumerBackoffDelay("PT10S"),
+								NewConsumerBackoffDelay("PT0.3S"),
 								NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 							),
 						),
@@ -773,7 +773,7 @@ func TestReconcileKind(t *testing.T) {
 								internals.Ordered,
 								NewConsumerTimeout("PT600S"),
 								NewConsumerRetry(10),
-								NewConsumerBackoffDelay("PT10S"),
+								NewConsumerBackoffDelay("PT0.3S"),
 								NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 							),
 						),
@@ -818,7 +818,7 @@ func TestReconcileKind(t *testing.T) {
 								internals.Ordered,
 								NewConsumerTimeout("PT600S"),
 								NewConsumerRetry(10),
-								NewConsumerBackoffDelay("PT10S"),
+								NewConsumerBackoffDelay("PT0.3S"),
 								NewConsumerBackoffPolicy(eventingduck.BackoffPolicyExponential),
 							),
 						),