Repository: kumahq/kuma
Pull request #2432 (2021-07-26 14:46:18)

fix(kuma-cp) set better keep-alive for bootstrap
### Summary

In some cases, when zone-ingress pod in kubernetes was restarted in parallel with zone-cp and if it was very quickly recreated it was able to connect to old instance of zone-cp and as the default values for keep-alive stands that it will start sending keep-alive probes after 2h, so it was possible that our TCP connection will be in half-closed state.

### Testing

- [ ] Unit tests
- [ ] E2E tests
- [ ] Manual testing on Universal
- [x] Manual testing on Kubernetes 

### Backwards compatibility

- [x] Add `backport-to-stable` label if the code is backwards compatible. Otherwise, list breaking changes.


codecov-commenter: # [Codecov](https://codecov.io/gh/kumahq/kuma/pull/2432?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq) Report
> Merging [#2432](https://codecov.io/gh/kumahq/kuma/pull/2432?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq) (bc50a01) into [master](https://codecov.io/gh/kumahq/kuma/commit/a196de9f67e114f733ffe1a5b1a903d3b79471d2?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq) (a196de9) will **increase** coverage by `0.14%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/kumahq/kuma/pull/2432/graphs/tree.svg?width=650&height=150&src=pr&token=D7Yvqs6rwV&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq)](https://codecov.io/gh/kumahq/kuma/pull/2432?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq)

```diff
@@            Coverage Diff             @@
##           master    #2432      +/-   ##
==========================================
+ Coverage   52.43%   52.58%   +0.14%     
==========================================
  Files         875      877       +2     
  Lines       47833    47855      +22     
==========================================
+ Hits        25083    25166      +83     
+ Misses      20714    20640      -74     
- Partials     2036     2049      +13     
```


| [Impacted Files](https://codecov.io/gh/kumahq/kuma/pull/2432?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq) | Coverage Δ | |
|---|---|---|
| [pkg/core/resources/store/customizable\_store.go](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq#diff-cGtnL2NvcmUvcmVzb3VyY2VzL3N0b3JlL2N1c3RvbWl6YWJsZV9zdG9yZS5nbw==) | `77.77% <0.00%> (-11.12%)` | :arrow_down: |
| [pkg/kds/client/sink.go](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq#diff-cGtnL2tkcy9jbGllbnQvc2luay5nbw==) | `47.72% <0.00%> (-4.55%)` | :arrow_down: |
| [pkg/plugins/resources/memory/store.go](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq#diff-cGtnL3BsdWdpbnMvcmVzb3VyY2VzL21lbW9yeS9zdG9yZS5nbw==) | `77.71% <0.00%> (-4.35%)` | :arrow_down: |
| [pkg/mads/v1/client/client.go](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq#diff-cGtnL21hZHMvdjEvY2xpZW50L2NsaWVudC5nbw==) | `41.25% <0.00%> (-2.50%)` | :arrow_down: |
| [api/observability/v1/mads.pb.go](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq#diff-YXBpL29ic2VydmFiaWxpdHkvdjEvbWFkcy5wYi5nbw==) | `34.53% <0.00%> (-1.04%)` | :arrow_down: |
| [api/internal/util/proto/types.go](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq#diff-YXBpL2ludGVybmFsL3V0aWwvcHJvdG8vdHlwZXMuZ28=) | `100.00% <0.00%> (ø)` | |
| [api/internal/util/proto/proto.go](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq#diff-YXBpL2ludGVybmFsL3V0aWwvcHJvdG8vcHJvdG8uZ28=) | `70.00% <0.00%> (ø)` | |
| [pkg/xds/generator/direct\_access\_proxy\_generator.go](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq#diff-cGtnL3hkcy9nZW5lcmF0b3IvZGlyZWN0X2FjY2Vzc19wcm94eV9nZW5lcmF0b3IuZ28=) | `83.90% <0.00%> (+1.14%)` | :arrow_up: |
| [api/mesh/v1alpha1/dataplane\_helpers.go](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq#diff-YXBpL21lc2gvdjFhbHBoYTEvZGF0YXBsYW5lX2hlbHBlcnMuZ28=) | `80.59% <0.00%> (+1.68%)` | :arrow_up: |
| [pkg/core/resources/manager/cache.go](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq#diff-cGtnL2NvcmUvcmVzb3VyY2VzL21hbmFnZXIvY2FjaGUuZ28=) | `81.81% <0.00%> (+2.59%)` | :arrow_up: |
| ... and [6 more](https://codecov.io/gh/kumahq/kuma/pull/2432/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/kumahq/kuma/pull/2432?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/kumahq/kuma/pull/2432?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq). Last update [a196de9...bc50a01](https://codecov.io/gh/kumahq/kuma/pull/2432?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=kumahq).


Patch:
diff --git a/pkg/xds/bootstrap/template_v3.go b/pkg/xds/bootstrap/template_v3.go
index 7fa262d49..b79a51cea 100644
--- a/pkg/xds/bootstrap/template_v3.go
+++ b/pkg/xds/bootstrap/template_v3.go
@@ -116,7 +116,10 @@ static_resources:
     upstream_connection_options:
       # configure a TCP keep-alive to detect and reconnect to the admin
       # server in the event of a TCP socket half open connection
-      tcp_keepalive: {}
+      tcp_keepalive:
+        keepalive_probes: 3
+        keepalive_time: 10
+        keepalive_interval: 10
     load_assignment:
       cluster_name: access_log_sink
       endpoints:
@@ -133,7 +136,10 @@ static_resources:
     upstream_connection_options:
       # configure a TCP keep-alive to detect and reconnect to the admin
       # server in the event of a TCP socket half open connection
-      tcp_keepalive: {}
+      tcp_keepalive:
+        keepalive_probes: 3
+        keepalive_time: 10
+        keepalive_interval: 10
     load_assignment:
       cluster_name: ads_cluster
       endpoints:


Test patch:
diff --git a/pkg/xds/bootstrap/testdata/bootstrap.k8s.golden.yaml b/pkg/xds/bootstrap/testdata/bootstrap.k8s.golden.yaml
index 6e99130c3..2d6613d4b 100644
--- a/pkg/xds/bootstrap/testdata/bootstrap.k8s.golden.yaml
+++ b/pkg/xds/bootstrap/testdata/bootstrap.k8s.golden.yaml
@@ -60,7 +60,10 @@ staticResources:
     name: access_log_sink
     type: STATIC
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
   - connectTimeout: 1s
     http2ProtocolOptions: {}
     loadAssignment:
@@ -88,7 +91,10 @@ staticResources:
         sni: localhost
     type: STRICT_DNS
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
 statsConfig:
   statsTags:
   - regex: ^grpc\.((.+)\.)
diff --git a/pkg/xds/bootstrap/testdata/bootstrap.overridden.golden.yaml b/pkg/xds/bootstrap/testdata/bootstrap.overridden.golden.yaml
index 7a01a5354..8d1ab7c5b 100644
--- a/pkg/xds/bootstrap/testdata/bootstrap.overridden.golden.yaml
+++ b/pkg/xds/bootstrap/testdata/bootstrap.overridden.golden.yaml
@@ -67,7 +67,10 @@ staticResources:
     name: access_log_sink
     type: STATIC
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
   - connectTimeout: 1s
     http2ProtocolOptions: {}
     loadAssignment:
@@ -95,7 +98,10 @@ staticResources:
         sni: localhost
     type: STRICT_DNS
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
 statsConfig:
   statsTags:
   - regex: ^grpc\.((.+)\.)
diff --git a/pkg/xds/bootstrap/testdata/bootstrap.universal.golden.yaml b/pkg/xds/bootstrap/testdata/bootstrap.universal.golden.yaml
index d64783b65..cd095c74a 100644
--- a/pkg/xds/bootstrap/testdata/bootstrap.universal.golden.yaml
+++ b/pkg/xds/bootstrap/testdata/bootstrap.universal.golden.yaml
@@ -60,7 +60,10 @@ staticResources:
     name: access_log_sink
     type: STATIC
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
   - connectTimeout: 1s
     http2ProtocolOptions: {}
     loadAssignment:
@@ -88,7 +91,10 @@ staticResources:
         sni: localhost
     type: STRICT_DNS
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
 statsConfig:
   statsTags:
   - regex: ^grpc\.((.+)\.)
diff --git a/pkg/xds/bootstrap/testdata/generator.custom-config-minimal-request.golden.yaml b/pkg/xds/bootstrap/testdata/generator.custom-config-minimal-request.golden.yaml
index ceee99e20..22acd0280 100644
--- a/pkg/xds/bootstrap/testdata/generator.custom-config-minimal-request.golden.yaml
+++ b/pkg/xds/bootstrap/testdata/generator.custom-config-minimal-request.golden.yaml
@@ -60,7 +60,10 @@ staticResources:
     name: access_log_sink
     type: STATIC
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
   - connectTimeout: 2s
     http2ProtocolOptions: {}
     loadAssignment:
@@ -88,7 +91,10 @@ staticResources:
         sni: localhost
     type: STRICT_DNS
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
 statsConfig:
   statsTags:
   - regex: ^grpc\.((.+)\.)
diff --git a/pkg/xds/bootstrap/testdata/generator.custom-config.golden.yaml b/pkg/xds/bootstrap/testdata/generator.custom-config.golden.yaml
index cd56c4c84..fb8f5709f 100644
--- a/pkg/xds/bootstrap/testdata/generator.custom-config.golden.yaml
+++ b/pkg/xds/bootstrap/testdata/generator.custom-config.golden.yaml
@@ -74,7 +74,10 @@ staticResources:
     name: access_log_sink
     type: STATIC
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
   - connectTimeout: 2s
     http2ProtocolOptions: {}
     loadAssignment:
@@ -102,7 +105,10 @@ staticResources:
         sni: localhost
     type: STRICT_DNS
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
 statsConfig:
   statsTags:
   - regex: ^grpc\.((.+)\.)
diff --git a/pkg/xds/bootstrap/testdata/generator.default-config-minimal-request.golden.yaml b/pkg/xds/bootstrap/testdata/generator.default-config-minimal-request.golden.yaml
index 6324d5b8f..7950d50ff 100644
--- a/pkg/xds/bootstrap/testdata/generator.default-config-minimal-request.golden.yaml
+++ b/pkg/xds/bootstrap/testdata/generator.default-config-minimal-request.golden.yaml
@@ -53,7 +53,10 @@ staticResources:
     name: access_log_sink
     type: STATIC
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
   - connectTimeout: 1s
     http2ProtocolOptions: {}
     loadAssignment:
@@ -81,7 +84,10 @@ staticResources:
         sni: localhost
     type: STRICT_DNS
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
 statsConfig:
   statsTags:
   - regex: ^grpc\.((.+)\.)
diff --git a/pkg/xds/bootstrap/testdata/generator.default-config.golden.yaml b/pkg/xds/bootstrap/testdata/generator.default-config.golden.yaml
index 746680dc8..653a9a6b0 100644
--- a/pkg/xds/bootstrap/testdata/generator.default-config.golden.yaml
+++ b/pkg/xds/bootstrap/testdata/generator.default-config.golden.yaml
@@ -69,7 +69,10 @@ staticResources:
     name: access_log_sink
     type: STATIC
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
   - connectTimeout: 1s
     http2ProtocolOptions: {}
     loadAssignment:
@@ -97,7 +100,10 @@ staticResources:
         sni: localhost
     type: STRICT_DNS
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
 statsConfig:
   statsTags:
   - regex: ^grpc\.((.+)\.)
diff --git a/pkg/xds/bootstrap/testdata/generator.default-config.kubernetes.golden.yaml b/pkg/xds/bootstrap/testdata/generator.default-config.kubernetes.golden.yaml
index 5495efa1b..580188947 100644
--- a/pkg/xds/bootstrap/testdata/generator.default-config.kubernetes.golden.yaml
+++ b/pkg/xds/bootstrap/testdata/generator.default-config.kubernetes.golden.yaml
@@ -57,7 +57,10 @@ staticResources:
     name: access_log_sink
     type: STATIC
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
   - connectTimeout: 1s
     http2ProtocolOptions: {}
     loadAssignment:
@@ -85,7 +88,10 @@ staticResources:
         sni: localhost
     type: STRICT_DNS
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
 statsConfig:
   statsTags:
   - regex: ^grpc\.((.+)\.)
diff --git a/pkg/xds/bootstrap/testdata/generator.default-config.kubernetes.ipv6.golden.yaml b/pkg/xds/bootstrap/testdata/generator.default-config.kubernetes.ipv6.golden.yaml
index c8dd2fde5..6753a1bfa 100644
--- a/pkg/xds/bootstrap/testdata/generator.default-config.kubernetes.ipv6.golden.yaml
+++ b/pkg/xds/bootstrap/testdata/generator.default-config.kubernetes.ipv6.golden.yaml
@@ -57,7 +57,10 @@ staticResources:
     name: access_log_sink
     type: STATIC
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
   - connectTimeout: 1s
     http2ProtocolOptions: {}
     loadAssignment:
@@ -85,7 +88,10 @@ staticResources:
         sni: fd00:a123::1
     type: STATIC
     upstreamConnectionOptions:
-      tcpKeepalive: {}
+      tcpKeepalive:
+        keepaliveInterval: 10
+        keepaliveProbes: 3
+        keepaliveTime: 10
 statsConfig:
   statsTags:
   - regex: ^grpc\.((.+)\.)