Repository: istio/istio (go)

Issue:
Endpoint Shards Erroneously Remain after Pod Removed
### Bug Description

Hi There 👋 

I am experiencing a prolonged issue with network routing that sometimes appears after deleting and re-applying a specific deployment & service pair. I'm not 100% convinced that it's a bug, however just posting here as I don't know what else to try.

The main symptom is that a random proportion of incoming traffic to a service begins failing with TLS and connection errors. I am running a TLS passthrough setup, and assumed this was the service itself. However, after extensive debugging, I could not replicate the issue when connecting directly. This led me to believe that the problem may lie with my networking setup.

Curiously, in the past, restarting the entire cluster had fixed the problem. However, it re-appeared this afternoon, and I was able to narrow the cause down to the ingress gateway pod itself. It appears, as seen below via the envoy logs (**), that the ingress gateway is attempting to load balance upstream traffic between two IP addresses, with one failing immediately and the other successfully responding. However, this seems erroneous, as there there is only a single pod defined in any of my deployments. Looking at the cluster, it can be confirmed that there is only a single running pod associated with that service. The second appears to be the cluster ip of an old pod, which no longer exists.

![IstioIssue drawio](https://user-images.githubusercontent.com/14295288/146052709-932c0922-2d60-4b49-85a0-fa814bacfa88.png)

My knowledge here is shady, however I assume istio flattens `v1/Service`' into their destination pod cluster IPs, and thereafter routes directly? If so, it appears that the istiod daemon has keeps a 'ghost' memory of an old pod endpoint.

This is shown in an istio debug dump, where the old IP address can be found in the `endpointShardz` file alongside the new one one. I've attached this below.

I'm not sure where to turn to next, however I hope this may help in finding whether this is something on my end, or if it is indeed a bug. For now, a workaround is to restart `istiod` when the error occurs. It seems to sync up correctly thereafter. Please let me know if I can provide any additional info! :) 

### Failure Case
#### Ingress Gateway Envoy Logs
```
2021-12-14T16:20:26.436791Z	debug	envoy filter	tls inspector: new connection accepted
2021-12-14T16:20:26.445607Z	debug	envoy filter	tls:onServerName(), requestedServerName: registry.solar.redacted
2021-12-14T16:20:26.445693Z	debug	envoy filter	[C543] new tcp proxy session
2021-12-14T16:20:26.445732Z	debug	envoy filter	[C543] Creating connection to cluster outbound|5000||registry.solar.svc.cluster.local
2021-12-14T16:20:26.445766Z	debug	envoy misc	Allocating TCP conn pool
2021-12-14T16:20:26.445824Z	debug	envoy pool	trying to create new connection
2021-12-14T16:20:26.445838Z	debug	envoy pool	creating a new connection
** 2021-12-14T16:20:26.445920Z	debug	envoy connection	[C544] connecting to 10.42.0.155:5000
2021-12-14T16:20:26.445956Z	debug	envoy connection	[C544] connection in progress
2021-12-14T16:20:26.445969Z	debug	envoy conn_handler	[C543] new connection from 10.42.0.150:58945
2021-12-14T16:20:26.769445Z	debug	envoy main	flushing stats
2021-12-14T16:20:27.069931Z	debug	envoy connection	[C544] delayed connect error: 113
2021-12-14T16:20:27.069962Z	debug	envoy connection	[C544] closing socket: 0
2021-12-14T16:20:27.069981Z	debug	envoy connection	[C544] TLS error: 268435588:SSL routines:OPENSSL_internal:CLIENTHELLO_TLSEXT 268435646:SSL routines:OPENSSL_internal:PARSE_TLSEXT
2021-12-14T16:20:27.070013Z	error	envoy envoy_bug	envoy bug failure: !(!failure_reason_.empty() && !transport_socket_->failureReason().empty()). Details: both connection and transport failure reason are not empty
2021-12-14T16:20:27.070026Z	debug	envoy pool	[C544] client disconnected, failure reason: delayed connect error: 113
2021-12-14T16:20:27.070043Z	debug	envoy filter	[C543] Creating connection to cluster outbound|5000||registry.solar.svc.cluster.local
2021-12-14T16:20:27.070054Z	debug	envoy connection	[C543] closing data_to_write=0 type=1
2021-12-14T16:20:27.070058Z	debug	envoy connection	[C543] closing socket: 1
```

### Success Case
#### Ingress Gateway Envoy Logs
```
2021-12-14T16:20:10.543809Z	debug	envoy filter	tls inspector: new connection accepted
2021-12-14T16:20:10.552654Z	debug	envoy filter	tls:onServerName(), requestedServerName: registry.solar.redacted
2021-12-14T16:20:10.552732Z	debug	envoy filter	[C521] new tcp proxy session
2021-12-14T16:20:10.552757Z	debug	envoy filter	[C521] Creating connection to cluster outbound|5000||registry.solar.svc.cluster.local
2021-12-14T16:20:10.552786Z	debug	envoy misc	Allocating TCP conn pool
2021-12-14T16:20:10.552801Z	debug	envoy pool	trying to create new connection
2021-12-14T16:20:10.552810Z	debug	envoy pool	creating a new connection
** 2021-12-14T16:20:10.552864Z	debug	envoy connection	[C522] connecting to 10.42.0.211:5000
2021-12-14T16:20:10.552988Z	debug	envoy connection	[C522] connection in progress
2021-12-14T16:20:10.553000Z	debug	envoy conn_handler	[C521] new connection from 10.42.0.150:58830
2021-12-14T16:20:10.553016Z	debug	envoy connection	[C522] connected
2021-12-14T16:20:10.553504Z	debug	envoy pool	[C522] attaching to next stream
2021-12-14T16:20:10.553522Z	debug	envoy pool	[C522] creating stream
2021-12-14T16:20:10.553534Z	debug	envoy filter	[C521] TCP:onUpstreamEvent(), requestedServerName: registry.solar.redacted
2021-12-14T16:20:11.020084Z	debug	envoy connection	[C521] remote close
2021-12-14T16:20:11.020123Z	debug	envoy connection	[C521] closing socket: 0
```

### Version

```prose
➜  ~ istioctl version
client version: 1.11.4
control plane version: 1.13-alpha.3b40614a3d801aedb79d7fd78735ac09a86073a7
data plane version: 1.13-alpha.3b40614a3d801aedb79d7fd78735ac09a86073a7 (8 proxies)

➜  ~ kubectl version --short
Client Version: v1.22.4
Server Version: v1.21.7+k3s1

➜  ~ helm version --short
v3.7.1+g1d11fcb
```


### Additional Information

<details closed>
<summary>manifest.yaml</summary>

```
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: registry-ingress
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 443
        name: https
        protocol: HTTPS
      hosts:
        - "registry.solar.redacted"
      tls:
        mode: PASSTHROUGH
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: registry-virtual-service
spec:
  hosts:
    - "registry.solar.redacted"
  gateways:
    - registry-ingress
  tls:
    - match:
      - port: 443
        sniHosts:
          - "registry.solar.redacted"
      route:
        - destination:
            host: registry
            port:
              number: 5000
---
apiVersion: v1
kind: Service
metadata:
  name: registry
  labels:
    app: registry
    service: registry
spec:
  ports:
    - port: 5000
      name: https
  selector:
    app: registry
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  labels:
    app: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
    spec:
      volumes:
        - <redacted>
      containers:
        - image: registry:2
          name: registry
          imagePullPolicy: IfNotPresent
          env:
            <redacted>
          ports:
            - containerPort: 5000s
          volumeMounts:
            <redacted>
```
</details>

<details closed>
<summary>endpointz</summary>

```
[
    {
        "svc": "registry.solar.svc.cluster.local:https",
        "ep": [
            {
                "service": {
                    "Attributes": {
                        "ServiceRegistry": "Kubernetes",
                        "Name": "registry",
                        "Namespace": "solar",
                        "Labels": {
                            "app": "registry",
                            "app.kubernetes.io/instance": "registry",
                            "service": "registry"
                        },
                        "ExportTo": null,
                        "LabelSelectors": {
                            "app": "registry"
                        },
                        "ClusterExternalAddresses": {
                            "Addresses": null
                        },
                        "ClusterExternalPorts": null
                    },
                    "ports": [
                        {
                            "name": "https",
                            "port": 5000,
                            "protocol": "HTTPS"
                        }
                    ],
                    "creationTime": "2021-12-14T12:31:50Z",
                    "hostname": "registry.solar.svc.cluster.local",
                    "clusterVIPs": {
                        "Addresses": {
                            "Kubernetes": [
                                "10.43.194.213"
                            ]
                        }
                    },
                    "defaultAddress": "10.43.194.213",
                    "Resolution": 0,
                    "MeshExternal": false,
                    "ResourceVersion": "309858"
                },
                "servicePort": {
                    "name": "https",
                    "port": 5000,
                    "protocol": "HTTPS"
                },
                "endpoint": {
                    "Labels": {
                        "app": "registry",
                        "istio-locality": "",
                        "pod-template-hash": "5b548db5",
                        "security.istio.io/tlsMode": "istio",
                        "service.istio.io/canonical-name": "registry",
                        "service.istio.io/canonical-revision": "latest",
                        "topology.istio.io/cluster": "Kubernetes",
                        "topology.istio.io/network": ""
                    },
                    "Address": "10.42.0.211",
                    "ServicePortName": "https",
                    "EnvoyEndpoint": null,
                    "ServiceAccount": "spiffe://cluster.local/ns/solar/sa/default",
                    "Network": "",
                    "Locality": {
                        "Label": "",
                        "ClusterID": "Kubernetes"
                    },
                    "EndpointPort": 5000,
                    "LbWeight": 0,
                    "TLSMode": "istio",
                    "Namespace": "solar",
                    "WorkloadName": "registry",
                    "HostName": "",
                    "SubDomain": "",
                    "TunnelAbility": 0
                }
            }
        ]
    }
]
```
</details>


<details closed>
<summary>endpointShardz</summary>

```
{
  "registry.solar.svc.cluster.local": {
    "solar": {
      "Shards": {
        "Kubernetes/Kubernetes": [
          {
            "Labels": {
              "app": "registry",
              "istio-locality": "",
              "pod-template-hash": "68c47fd55c",
              "security.istio.io/tlsMode": "istio",
              "service.istio.io/canonical-name": "registry",
              "service.istio.io/canonical-revision": "latest",
              "topology.istio.io/cluster": "Kubernetes",
              "topology.istio.io/network": ""
            },
            "Address": "10.42.0.155",
            "ServicePortName": "https",
            "EnvoyEndpoint": {
              "HostIdentifier": {
                "Endpoint": {
                  "address": {
                    "Address": {
                      "SocketAddress": {
                        "address": "10.42.0.155",
                        "PortSpecifier": {
                          "PortValue": 5000
                        }
                      }
                    }
                  }
                }
              },
              "metadata": {
                "filter_metadata": {
                  "envoy.transport_socket_match": {
                    "tlsMode": "istio"
                  },
                  "istio": {
                    "workload": "registry;solar;registry;latest;Kubernetes"
                  }
                }
              },
              "load_balancing_weight": {
                "value": 1
              }
            },
            "ServiceAccount": "spiffe://cluster.local/ns/solar/sa/default",
            "Network": "",
            "Locality": {
              "Label": "",
              "ClusterID": "Kubernetes"
            },
            "EndpointPort": 5000,
            "LbWeight": 0,
            "TLSMode": "istio",
            "Namespace": "solar",
            "WorkloadName": "registry",
            "HostName": "",
            "SubDomain": "",
            "TunnelAbility": 0
          },
          {
            "Labels": {
              "app": "registry",
              "istio-locality": "",
              "pod-template-hash": "5b548db5",
              "security.istio.io/tlsMode": "istio",
              "service.istio.io/canonical-name": "registry",
              "service.istio.io/canonical-revision": "latest",
              "topology.istio.io/cluster": "Kubernetes",
              "topology.istio.io/network": ""
            },
            "Address": "10.42.0.211",
            "ServicePortName": "https",
            "EnvoyEndpoint": {
              "HostIdentifier": {
                "Endpoint": {
                  "address": {
                    "Address": {
                      "SocketAddress": {
                        "address": "10.42.0.211",
                        "PortSpecifier": {
                          "PortValue": 5000
                        }
                      }
                    }
                  }
                }
              },
              "metadata": {
                "filter_metadata": {
                  "envoy.transport_socket_match": {
                    "tlsMode": "istio"
                  },
                  "istio": {
                    "workload": "registry;solar;registry;latest;Kubernetes"
                  }
                }
              },
              "load_balancing_weight": {
                "value": 1
              }
            },
            "ServiceAccount": "spiffe://cluster.local/ns/solar/sa/default",
            "Network": "",
            "Locality": {
              "Label": "",
              "ClusterID": "Kubernetes"
            },
            "EndpointPort": 5000,
            "LbWeight": 0,
            "TLSMode": "istio",
            "Namespace": "solar",
            "WorkloadName": "registry",
            "HostName": "",
            "SubDomain": "",
            "TunnelAbility": 0
          }
        ]
      },
      "ServiceAccounts": {
        "spiffe://cluster.local/ns/solar/sa/default": {}
      }
    }
  }
}
```
</details>


<details closed>
<summary>discovery [snippet]</summary>

```
2021-12-13T17:15:02.760343Z	info	Adding Kubernetes registry adapter
2021-12-13T17:15:02.983587Z	info	initializing registry event handlers
2021-12-13T17:15:02.986014Z	info	kube	Initializing Kubernetes service registry "Kubernetes"
2021-12-13T17:15:03.090380Z	info	ads	Full push, new service solar/registry.solar.svc.cluster.local
2021-12-13T17:15:03.090392Z	info	ads	Full push, service accounts changed, registry.solar.svc.cluster.local
2021-12-13T17:15:13.436987Z	info	ads	ADS: new connection for node:registry-6f54b48768-9lfdc.solar-2
2021-12-13T17:15:13.439067Z	info	ads	CDS: PUSH request for node:registry-6f54b48768-9lfdc.solar resources:35 size:30.6kB cached:0/30
2021-12-13T17:15:13.439290Z	info	ads	EDS: PUSH request for node:registry-6f54b48768-9lfdc.solar resources:29 size:4.6kB empty:0 cached:29/29
2021-12-13T17:15:13.446277Z	info	ads	LDS: PUSH request for node:registry-6f54b48768-9lfdc.solar resources:29 size:90.6kB
2021-12-13T17:15:13.447905Z	info	ads	RDS: PUSH request for node:registry-6f54b48768-9lfdc.solar resources:18 size:10.8kB cached:0/18
2021-12-13T17:15:13.499197Z	info	ads	EDS: PUSH request for node:registry-6f54b48768-9lfdc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T17:15:13.548612Z	info	ads	RDS: PUSH request for node:registry-6f54b48768-9lfdc.solar resources:19 size:11.5kB cached:18/19
2021-12-13T17:15:51.344433Z	info	ads	Push debounce stable[7] 1 for config Gateway/solar/registry-ingress: 100.180433ms since last change, 100.180164ms since last push, full=true
2021-12-13T17:15:51.657516Z	info	ads	Push debounce stable[8] 1 for config VirtualService/solar/registry-virtual-service: 100.251156ms since last change, 100.250737ms since last push, full=true
2021-12-13T17:48:06.661151Z	info	ads	ADS: "10.42.0.142:34140" registry-6f54b48768-9lfdc.solar-2 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T17:48:07.126873Z	info	ads	ADS: new connection for node:registry-6f54b48768-9lfdc.solar-4
2021-12-13T17:48:07.127442Z	info	ads	CDS: PUSH request for node:registry-6f54b48768-9lfdc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T17:48:07.127654Z	info	ads	EDS: PUSH request for node:registry-6f54b48768-9lfdc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T17:48:07.129865Z	info	ads	LDS: PUSH request for node:registry-6f54b48768-9lfdc.solar resources:29 size:90.6kB
2021-12-13T17:48:07.130070Z	info	ads	RDS: PUSH request for node:registry-6f54b48768-9lfdc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T17:53:32.728861Z	info	Sidecar injection request for solar/registry-fbbd65b99-***** (actual name not yet known)
2021-12-13T17:53:36.041095Z	info	ads	Push debounce stable[9] 1 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.255788ms since last change, 100.255304ms since last push, full=false
2021-12-13T17:53:42.694721Z	info	ads	ADS: new connection for node:registry-fbbd65b99-fmbw9.solar-5
2021-12-13T17:53:42.695924Z	info	ads	CDS: PUSH request for node:registry-fbbd65b99-fmbw9.solar resources:35 size:30.6kB cached:0/30
2021-12-13T17:53:42.714034Z	info	ads	EDS: PUSH request for node:registry-fbbd65b99-fmbw9.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T17:53:43.044787Z	info	ads	LDS: PUSH request for node:registry-fbbd65b99-fmbw9.solar resources:29 size:90.6kB
2021-12-13T17:53:43.096349Z	info	ads	RDS: PUSH request for node:registry-fbbd65b99-fmbw9.solar resources:19 size:11.5kB cached:0/19
2021-12-13T17:53:44.795585Z	info	ads	CDS: PUSH for node:registry-fbbd65b99-fmbw9.solar resources:35 size:30.6kB cached:30/30
2021-12-13T17:53:44.795783Z	info	ads	EDS: PUSH for node:registry-fbbd65b99-fmbw9.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T17:53:44.799228Z	info	ads	LDS: PUSH for node:registry-fbbd65b99-fmbw9.solar resources:29 size:90.6kB
2021-12-13T17:53:44.799631Z	info	ads	RDS: PUSH for node:registry-fbbd65b99-fmbw9.solar resources:19 size:11.5kB cached:19/19
2021-12-13T17:53:44.939165Z	info	ads	Push debounce stable[10] 2 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.222542ms since last change, 137.589451ms since last push, full=false
2021-12-13T17:53:49.853373Z	info	ads	ADS: "10.42.0.142:54370" registry-6f54b48768-9lfdc.solar-4 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T17:56:33.991275Z	info	Sidecar injection request for solar/registry-84fc86c745-***** (actual name not yet known)
2021-12-13T17:56:36.436451Z	info	ads	Push debounce stable[11] 1 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.187618ms since last change, 100.187262ms since last push, full=false
2021-12-13T17:56:44.090385Z	info	ads	ADS: new connection for node:registry-84fc86c745-k2zpg.solar-6
2021-12-13T17:56:44.090778Z	info	ads	CDS: PUSH request for node:registry-84fc86c745-k2zpg.solar resources:35 size:30.6kB cached:29/30
2021-12-13T17:56:44.112728Z	info	ads	EDS: PUSH request for node:registry-84fc86c745-k2zpg.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T17:56:44.243720Z	info	ads	LDS: PUSH request for node:registry-84fc86c745-k2zpg.solar resources:29 size:90.6kB
2021-12-13T17:56:44.292467Z	info	ads	RDS: PUSH request for node:registry-84fc86c745-k2zpg.solar resources:19 size:11.5kB cached:19/19
2021-12-13T17:56:46.055138Z	info	ads	CDS: PUSH for node:registry-84fc86c745-k2zpg.solar resources:35 size:30.6kB cached:30/30
2021-12-13T17:56:46.055363Z	info	ads	EDS: PUSH for node:registry-84fc86c745-k2zpg.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T17:56:46.058787Z	info	ads	LDS: PUSH for node:registry-84fc86c745-k2zpg.solar resources:29 size:90.6kB
2021-12-13T17:56:46.059119Z	info	ads	RDS: PUSH for node:registry-84fc86c745-k2zpg.solar resources:19 size:11.5kB cached:19/19
2021-12-13T17:56:46.186228Z	info	ads	Push debounce stable[12] 2 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.239262ms since last change, 121.976953ms since last push, full=false
2021-12-13T17:56:51.091554Z	info	ads	ADS: "10.42.0.151:41120" registry-fbbd65b99-fmbw9.solar-5 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T18:15:13.892485Z	info	Sidecar injection request for solar/registry-d95f867b8-***** (actual name not yet known)
2021-12-13T18:15:16.916218Z	info	ads	Push debounce stable[13] 1 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.240603ms since last change, 100.240245ms since last push, full=false
2021-12-13T18:15:23.496241Z	info	ads	ADS: new connection for node:registry-d95f867b8-qb99q.solar-7
2021-12-13T18:15:23.496641Z	info	ads	CDS: PUSH request for node:registry-d95f867b8-qb99q.solar resources:35 size:30.6kB cached:29/30
2021-12-13T18:15:23.512099Z	info	ads	EDS: PUSH request for node:registry-d95f867b8-qb99q.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T18:15:23.662156Z	info	ads	LDS: PUSH request for node:registry-d95f867b8-qb99q.solar resources:29 size:90.6kB
2021-12-13T18:15:23.710230Z	info	ads	RDS: PUSH request for node:registry-d95f867b8-qb99q.solar resources:19 size:11.5kB cached:19/19
2021-12-13T18:15:23.953653Z	info	ads	CDS: PUSH for node:registry-d95f867b8-qb99q.solar resources:35 size:30.6kB cached:30/30
2021-12-13T18:15:23.953924Z	info	ads	EDS: PUSH for node:registry-d95f867b8-qb99q.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T18:15:23.957481Z	info	ads	LDS: PUSH for node:registry-d95f867b8-qb99q.solar resources:29 size:90.6kB
2021-12-13T18:15:23.957775Z	info	ads	RDS: PUSH for node:registry-d95f867b8-qb99q.solar resources:19 size:11.5kB cached:19/19
2021-12-13T18:15:24.106897Z	info	ads	Push debounce stable[14] 2 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.745161ms since last change, 147.028559ms since last push, full=false
2021-12-13T18:15:29.014082Z	info	ads	ADS: "10.42.0.152:37628" registry-84fc86c745-k2zpg.solar-6 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T18:17:07.477780Z	info	Sidecar injection request for solar/registry-6fccb7b775-***** (actual name not yet known)
2021-12-13T18:17:10.260328Z	info	ads	Push debounce stable[15] 1 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.246118ms since last change, 100.245812ms since last push, full=false
2021-12-13T18:17:16.753600Z	info	ads	ADS: new connection for node:registry-6fccb7b775-jrsh8.solar-9
2021-12-13T18:17:16.753992Z	info	ads	CDS: PUSH request for node:registry-6fccb7b775-jrsh8.solar resources:35 size:30.6kB cached:29/30
2021-12-13T18:17:16.768782Z	info	ads	EDS: PUSH request for node:registry-6fccb7b775-jrsh8.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T18:17:16.803296Z	info	ads	LDS: PUSH request for node:registry-6fccb7b775-jrsh8.solar resources:29 size:90.6kB
2021-12-13T18:17:16.855117Z	info	ads	RDS: PUSH request for node:registry-6fccb7b775-jrsh8.solar resources:19 size:11.5kB cached:19/19
2021-12-13T18:17:17.546001Z	info	ads	CDS: PUSH for node:registry-6fccb7b775-jrsh8.solar resources:35 size:30.6kB cached:30/30
2021-12-13T18:17:17.546262Z	info	ads	EDS: PUSH for node:registry-6fccb7b775-jrsh8.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T18:17:17.551155Z	info	ads	LDS: PUSH for node:registry-6fccb7b775-jrsh8.solar resources:29 size:90.6kB
2021-12-13T18:17:17.551499Z	info	ads	RDS: PUSH for node:registry-6fccb7b775-jrsh8.solar resources:19 size:11.5kB cached:19/19
2021-12-13T18:17:17.677055Z	info	ads	Push debounce stable[16] 2 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.40358ms since last change, 120.128848ms since last push, full=false
2021-12-13T18:17:22.603604Z	info	ads	ADS: "10.42.0.153:40412" registry-d95f867b8-qb99q.solar-7 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T18:17:45.536973Z	info	Sidecar injection request for solar/registry-68c47fd55c-***** (actual name not yet known)
2021-12-13T18:17:48.370634Z	info	ads	Push debounce stable[17] 1 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.219796ms since last change, 100.219427ms since last push, full=false
2021-12-13T18:17:55.063622Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-10
2021-12-13T18:17:55.063982Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:29/30
2021-12-13T18:17:55.081120Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T18:17:55.348417Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T18:17:55.396925Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T18:17:55.591851Z	info	ads	CDS: PUSH for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T18:17:55.592006Z	info	ads	EDS: PUSH for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T18:17:55.594757Z	info	ads	LDS: PUSH for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T18:17:55.595051Z	info	ads	RDS: PUSH for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T18:17:55.731175Z	info	ads	Push debounce stable[18] 2 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.346681ms since last change, 130.702447ms since last push, full=false
2021-12-13T18:18:00.644173Z	info	ads	ADS: "10.42.0.154:51418" registry-6fccb7b775-jrsh8.solar-9 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T18:46:03.974124Z	info	ads	ADS: "10.42.0.155:46832" registry-68c47fd55c-c7hlc.solar-10 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T18:46:04.190742Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-12
2021-12-13T18:46:04.191450Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:29/30
2021-12-13T18:46:04.191632Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T18:46:04.195414Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T18:46:04.195656Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T19:17:57.916238Z	info	ads	ADS: "10.42.0.155:40036" registry-68c47fd55c-c7hlc.solar-12 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T19:17:58.260775Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-14
2021-12-13T19:17:58.261397Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T19:17:58.261667Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T19:17:58.265346Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T19:17:58.265624Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T19:46:27.122495Z	info	ads	ADS: "10.42.0.155:59732" registry-68c47fd55c-c7hlc.solar-14 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T19:46:27.363410Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-15
2021-12-13T19:46:27.364001Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T19:46:27.364277Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T19:46:27.368065Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T19:46:27.368408Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T20:14:08.249070Z	info	ads	ADS: "10.42.0.155:48748" registry-68c47fd55c-c7hlc.solar-15 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T20:14:08.555104Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-17
2021-12-13T20:14:08.555712Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T20:14:08.555958Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T20:14:08.559809Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T20:14:08.560333Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T20:42:27.247422Z	info	ads	ADS: "10.42.0.155:37574" registry-68c47fd55c-c7hlc.solar-17 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T20:42:27.404331Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-19
2021-12-13T20:42:27.404822Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T20:42:27.405044Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T20:42:27.408017Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T20:42:27.408285Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T21:14:40.723556Z	info	ads	ADS: "10.42.0.155:38338" registry-68c47fd55c-c7hlc.solar-19 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T21:14:41.188897Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-21
2021-12-13T21:14:41.189463Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T21:14:41.189876Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T21:14:41.193384Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T21:14:41.193734Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T21:46:21.830424Z	info	ads	ADS: "10.42.0.155:58346" registry-68c47fd55c-c7hlc.solar-21 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T21:46:22.242834Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-23
2021-12-13T21:46:22.243330Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T21:46:22.243536Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T21:46:22.246922Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T21:46:22.247382Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T22:15:26.769776Z	info	ads	ADS: "10.42.0.155:49422" registry-68c47fd55c-c7hlc.solar-23 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T22:15:26.874593Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-25
2021-12-13T22:15:26.874970Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T22:15:26.875143Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T22:15:26.877203Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T22:15:26.877477Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T22:47:38.069165Z	info	ads	ADS: "10.42.0.155:39800" registry-68c47fd55c-c7hlc.solar-25 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T22:47:38.552935Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-27
2021-12-13T22:47:38.553249Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T22:47:38.553363Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T22:47:38.555281Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T22:47:38.555478Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T23:17:04.292403Z	info	ads	ADS: "10.42.0.155:59404" registry-68c47fd55c-c7hlc.solar-27 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T23:17:04.557608Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-29
2021-12-13T23:17:04.558139Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T23:17:04.558331Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T23:17:04.561576Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T23:17:04.561940Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-13T23:49:55.297452Z	info	ads	ADS: "10.42.0.155:49326" registry-68c47fd55c-c7hlc.solar-29 terminated rpc error: code = Canceled desc = context canceled
2021-12-13T23:49:55.634196Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-31
2021-12-13T23:49:55.634626Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-13T23:49:55.634795Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-13T23:49:55.637434Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-13T23:49:55.637733Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T00:21:09.835695Z	info	ads	ADS: "10.42.0.155:41118" registry-68c47fd55c-c7hlc.solar-31 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T00:21:09.970410Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-34
2021-12-14T00:21:09.970735Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T00:21:09.970914Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T00:21:09.972718Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T00:21:09.972898Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T00:49:14.161422Z	info	ads	ADS: "10.42.0.155:60074" registry-68c47fd55c-c7hlc.solar-34 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T00:49:14.502631Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-36
2021-12-14T00:49:14.503188Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T00:49:14.503428Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T00:49:14.506604Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T00:49:14.506780Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T01:18:32.267115Z	info	ads	ADS: "10.42.0.155:48958" registry-68c47fd55c-c7hlc.solar-36 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T01:18:32.445723Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-37
2021-12-14T01:18:32.446169Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T01:18:32.446346Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T01:18:32.448826Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T01:18:32.449091Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T01:51:25.394543Z	info	ads	ADS: "10.42.0.155:38586" registry-68c47fd55c-c7hlc.solar-37 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T01:51:25.564130Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-39
2021-12-14T01:51:25.564628Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T01:51:25.564896Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T01:51:25.568376Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T01:51:25.568767Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T02:22:44.031974Z	info	ads	ADS: "10.42.0.155:58578" registry-68c47fd55c-c7hlc.solar-39 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T02:22:44.262529Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-42
2021-12-14T02:22:44.263085Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T02:22:44.263352Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T02:22:44.266784Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T02:22:44.267094Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T02:53:10.309380Z	info	ads	ADS: "10.42.0.155:49446" registry-68c47fd55c-c7hlc.solar-42 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T02:53:10.732707Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-44
2021-12-14T02:53:10.733188Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T02:53:10.733401Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T02:53:10.738225Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T02:53:10.738530Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T03:20:54.444551Z	info	ads	ADS: "10.42.0.155:40032" registry-68c47fd55c-c7hlc.solar-44 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T03:20:54.561180Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-46
2021-12-14T03:20:54.561757Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T03:20:54.562091Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T03:20:54.565788Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T03:20:54.566165Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T03:48:44.575474Z	info	ads	ADS: "10.42.0.155:56804" registry-68c47fd55c-c7hlc.solar-46 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T03:48:44.776936Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-47
2021-12-14T03:48:44.777578Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T03:48:44.777776Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T03:48:44.780363Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T03:48:44.780574Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T04:17:29.189724Z	info	ads	ADS: "10.42.0.155:45740" registry-68c47fd55c-c7hlc.solar-47 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T04:17:29.468532Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-49
2021-12-14T04:17:29.469149Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T04:17:29.469453Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T04:17:29.474623Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T04:17:29.475041Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T04:50:13.236391Z	info	ads	ADS: "10.42.0.155:35098" registry-68c47fd55c-c7hlc.solar-49 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T04:50:13.382713Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-52
2021-12-14T04:50:13.383313Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T04:50:13.383551Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T04:50:13.385542Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T04:50:13.385771Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T05:22:49.865830Z	info	ads	ADS: "10.42.0.155:55112" registry-68c47fd55c-c7hlc.solar-52 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T05:22:50.203077Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-54
2021-12-14T05:22:50.203394Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T05:22:50.203514Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T05:22:50.206620Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T05:22:50.206904Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T05:54:52.563432Z	info	ads	ADS: "10.42.0.155:46836" registry-68c47fd55c-c7hlc.solar-54 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T05:54:52.968714Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-56
2021-12-14T05:54:52.969420Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T05:54:52.969732Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T05:54:52.973998Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T05:54:52.974377Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T06:24:16.873569Z	info	ads	ADS: "10.42.0.155:38100" registry-68c47fd55c-c7hlc.solar-56 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T06:24:16.921280Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-58
2021-12-14T06:24:16.921906Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T06:24:16.922270Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T06:24:16.924430Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T06:24:16.924730Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T06:54:31.972785Z	info	ads	ADS: "10.42.0.155:55916" registry-68c47fd55c-c7hlc.solar-58 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T06:54:32.396630Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-60
2021-12-14T06:54:32.397314Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T06:54:32.397660Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T06:54:32.402107Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T06:54:32.402478Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T07:24:58.014254Z	info	ads	ADS: "10.42.0.155:45800" registry-68c47fd55c-c7hlc.solar-60 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T07:24:58.052223Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-61
2021-12-14T07:24:58.052824Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T07:24:58.053148Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T07:24:58.056652Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T07:24:58.057015Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T07:53:39.147134Z	info	ads	ADS: "10.42.0.155:35728" registry-68c47fd55c-c7hlc.solar-61 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T07:53:39.613987Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-63
2021-12-14T07:53:39.614633Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T07:53:39.614941Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T07:53:39.618708Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T07:53:39.619085Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T08:21:23.137936Z	info	ads	ADS: "10.42.0.155:53016" registry-68c47fd55c-c7hlc.solar-63 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T08:21:23.452980Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-65
2021-12-14T08:21:23.453564Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T08:21:23.453885Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T08:21:23.457966Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T08:21:23.458154Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T08:54:21.492006Z	info	ads	ADS: "10.42.0.155:41370" registry-68c47fd55c-c7hlc.solar-65 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T08:54:21.607308Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-67
2021-12-14T08:54:21.607912Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T08:54:21.608223Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T08:54:21.610418Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T08:54:21.610743Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T09:27:09.508170Z	info	ads	ADS: "10.42.0.155:33488" registry-68c47fd55c-c7hlc.solar-67 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T09:27:10.018494Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-70
2021-12-14T09:27:10.019079Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T09:27:10.019290Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T09:27:10.022575Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T09:27:10.023006Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T09:59:17.520670Z	info	ads	ADS: "10.42.0.155:53072" registry-68c47fd55c-c7hlc.solar-70 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T09:59:18.012103Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-72
2021-12-14T09:59:18.012712Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T09:59:18.012992Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T09:59:18.016812Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T09:59:18.017092Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T10:31:17.340149Z	info	ads	ADS: "10.42.0.155:44358" registry-68c47fd55c-c7hlc.solar-72 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T10:31:17.512779Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-74
2021-12-14T10:31:17.513284Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T10:31:17.513515Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T10:31:17.517540Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T10:31:17.517777Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T11:00:06.893932Z	info	ads	ADS: "10.42.0.155:35160" registry-68c47fd55c-c7hlc.solar-74 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T11:00:07.069137Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-75
2021-12-14T11:00:07.069457Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T11:00:07.069595Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T11:00:07.071566Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T11:00:07.071761Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T11:27:57.917477Z	info	ads	ADS: "10.42.0.155:52630" registry-68c47fd55c-c7hlc.solar-75 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T11:27:58.239070Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-77
2021-12-14T11:27:58.239699Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T11:27:58.240105Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T11:27:58.243685Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T11:27:58.244213Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T11:59:13.864427Z	info	ads	ADS: "10.42.0.155:41396" registry-68c47fd55c-c7hlc.solar-77 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T11:59:13.990983Z	info	ads	ADS: new connection for node:registry-68c47fd55c-c7hlc.solar-79
2021-12-14T11:59:13.991659Z	info	ads	CDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:35 size:30.6kB cached:30/30
2021-12-14T11:59:13.992000Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T11:59:13.995981Z	info	ads	LDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:90.6kB
2021-12-14T11:59:13.996394Z	info	ads	RDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T12:29:16.295810Z	info	ads	Incremental push, service registry.solar.svc.cluster.local has no endpoints
2021-12-14T12:29:16.396244Z	info	ads	Push debounce stable[19] 2 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.332332ms since last change, 115.796614ms since last push, full=true
2021-12-14T12:29:16.399229Z	info	ads	CDS: PUSH for node:registry-68c47fd55c-c7hlc.solar resources:33 size:29.5kB cached:29/29
2021-12-14T12:29:16.399695Z	info	ads	EDS: PUSH for node:registry-68c47fd55c-c7hlc.solar resources:30 size:4.6kB empty:1 cached:29/30
2021-12-14T12:29:16.415535Z	info	ads	LDS: PUSH for node:registry-68c47fd55c-c7hlc.solar resources:28 size:86.5kB
2021-12-14T12:29:16.415861Z	info	ads	RDS: PUSH for node:registry-68c47fd55c-c7hlc.solar resources:19 size:11.5kB cached:19/19
2021-12-14T12:29:16.455877Z	info	ads	EDS: PUSH request for node:registry-68c47fd55c-c7hlc.solar resources:29 size:4.6kB empty:0 cached:29/29
2021-12-14T12:29:16.705878Z	info	ads	Push debounce stable[20] 1 for config Gateway/solar/registry-ingress: 100.31883ms since last change, 100.318572ms since last push, full=true
2021-12-14T12:29:16.866359Z	info	ads	Push debounce stable[21] 1 for config VirtualService/solar/registry-virtual-service: 100.275432ms since last change, 100.274938ms since last push, full=true
2021-12-14T12:29:21.710443Z	info	ads	ADS: "10.42.0.155:60314" registry-68c47fd55c-c7hlc.solar-79 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T12:31:50.723929Z	info	ads	Full push, new service solar/registry.solar.svc.cluster.local
2021-12-14T12:31:50.723962Z	info	ads	Full push, service accounts changed, registry.solar.svc.cluster.local
2021-12-14T12:31:50.843548Z	info	ads	Push debounce stable[22] 4 for config ServiceEntry/solar/registry.solar.svc.cluster.local and 2 more configs: 100.313348ms since last change, 139.365161ms since last push, full=true
2021-12-14T12:33:16.126252Z	info	Sidecar injection request for solar/registry-8644dd67c8-***** (actual name not yet known)
2021-12-14T12:33:16.347956Z	info	ads	Push debounce stable[23] 1 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.63569ms since last change, 100.635472ms since last push, full=false
2021-12-14T12:33:19.362683Z	info	ads	Push debounce stable[24] 1 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.097236ms since last change, 100.096812ms since last push, full=false
2021-12-14T12:33:27.146393Z	info	ads	ADS: new connection for node:registry-8644dd67c8-g7b28.solar-81
2021-12-14T12:33:27.146725Z	info	ads	CDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:35 size:30.5kB cached:29/30
2021-12-14T12:33:27.166881Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T12:33:27.472747Z	info	ads	LDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:29 size:90.6kB
2021-12-14T12:33:27.528539Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:19 size:11.5kB cached:19/19
2021-12-14T12:33:28.282915Z	info	ads	CDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:35 size:30.5kB cached:30/30
2021-12-14T12:33:28.283117Z	info	ads	EDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:30 size:4.8kB empty:0 cached:30/30
2021-12-14T12:33:28.286424Z	info	ads	LDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:29 size:90.6kB
2021-12-14T12:33:28.286714Z	info	ads	RDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:19 size:11.5kB cached:19/19
2021-12-14T12:33:28.393413Z	info	ads	Push debounce stable[25] 1 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.263414ms since last change, 100.263053ms since last push, full=false
2021-12-14T12:56:45.266946Z	info	ads	CDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:37 size:32.3kB cached:29/32
2021-12-14T12:56:45.267248Z	info	ads	EDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:30 size:4.9kB empty:0 cached:30/30
2021-12-14T12:56:45.274047Z	info	ads	LDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:31 size:96.9kB
2021-12-14T12:56:45.274493Z	info	ads	RDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:19 size:11.5kB cached:19/19
2021-12-14T12:56:45.283726Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:32 size:5.1kB empty:2 cached:30/32
2021-12-14T12:56:45.294392Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:19/21
2021-12-14T12:57:02.956573Z	info	ads	CDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:35 size:30.6kB cached:30/30
2021-12-14T12:57:02.956951Z	info	ads	EDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:32 size:5.1kB empty:2 cached:30/32
2021-12-14T12:57:02.959979Z	info	ads	LDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:29 size:90.6kB
2021-12-14T12:57:02.960319Z	info	ads	RDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:21 size:11.6kB cached:19/21
2021-12-14T12:57:02.964872Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:30 size:4.9kB empty:0 cached:30/30
2021-12-14T12:57:12.693027Z	info	ads	CDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:37 size:32.3kB cached:30/32
2021-12-14T12:57:12.693725Z	info	ads	EDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:30 size:4.9kB empty:0 cached:30/30
2021-12-14T12:57:12.700993Z	info	ads	LDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:31 size:96.9kB
2021-12-14T12:57:12.701281Z	info	ads	RDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T12:57:12.721740Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:32 size:5.1kB empty:0 cached:32/32
2021-12-14T12:57:41.520158Z	info	ads	CDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:37 size:32.4kB cached:30/32
2021-12-14T12:57:41.520673Z	info	ads	EDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:32 size:5.4kB empty:0 cached:31/32
2021-12-14T12:57:41.523874Z	info	ads	LDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:31 size:96.9kB
2021-12-14T12:57:41.524086Z	info	ads	RDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T12:57:49.542703Z	info	ads	CDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:37 size:32.4kB cached:31/32
2021-12-14T12:57:49.543261Z	info	ads	EDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:32 size:5.5kB empty:0 cached:31/32
2021-12-14T12:57:49.546418Z	info	ads	LDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:31 size:96.9kB
2021-12-14T12:57:49.546743Z	info	ads	RDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T13:01:26.122794Z	info	ads	ADS: "10.42.0.156:49186" registry-8644dd67c8-g7b28.solar-81 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T13:01:26.198687Z	info	ads	ADS: new connection for node:registry-8644dd67c8-g7b28.solar-88
2021-12-14T13:01:26.199369Z	info	ads	CDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:37 size:32.4kB cached:31/32
2021-12-14T13:01:26.199670Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:32 size:5.8kB empty:0 cached:32/32
2021-12-14T13:01:26.203655Z	info	ads	LDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:31 size:96.9kB
2021-12-14T13:01:26.204062Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T13:24:41.959662Z	info	ads	CDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:38 size:33.3kB cached:33/33
2021-12-14T13:24:41.960642Z	info	ads	EDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:32 size:5.8kB empty:0 cached:32/32
2021-12-14T13:24:41.965750Z	info	ads	LDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:32 size:97.6kB
2021-12-14T13:24:41.970090Z	info	ads	RDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T13:24:41.991346Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:33 size:5.8kB empty:0 cached:33/33
2021-12-14T13:30:21.191057Z	info	ads	ADS: "10.42.0.156:39406" registry-8644dd67c8-g7b28.solar-88 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T13:30:21.251779Z	info	ads	ADS: new connection for node:registry-8644dd67c8-g7b28.solar-94
2021-12-14T13:30:21.252219Z	info	ads	CDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:38 size:33.3kB cached:33/33
2021-12-14T13:30:21.252416Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:33 size:5.8kB empty:0 cached:33/33
2021-12-14T13:30:21.255314Z	info	ads	LDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:32 size:97.6kB
2021-12-14T13:30:21.255678Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T13:32:54.553732Z	info	ads	CDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:38 size:33.4kB cached:33/33
2021-12-14T13:32:54.571045Z	info	ads	EDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:33 size:5.8kB empty:0 cached:32/33
2021-12-14T13:32:54.588683Z	info	ads	LDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:32 size:97.6kB
2021-12-14T13:32:54.591290Z	info	ads	RDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T14:02:19.609573Z	info	ads	ADS: "10.42.0.156:39030" registry-8644dd67c8-g7b28.solar-94 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T14:02:19.676197Z	info	ads	ADS: new connection for node:registry-8644dd67c8-g7b28.solar-114
2021-12-14T14:02:19.676850Z	info	ads	CDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:38 size:33.4kB cached:33/33
2021-12-14T14:02:19.677205Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:33 size:5.8kB empty:0 cached:33/33
2021-12-14T14:02:19.681978Z	info	ads	LDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:32 size:97.6kB
2021-12-14T14:02:19.682336Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T14:33:41.060210Z	info	ads	ADS: "10.42.0.156:44466" registry-8644dd67c8-g7b28.solar-114 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T14:33:41.218879Z	info	ads	ADS: new connection for node:registry-8644dd67c8-g7b28.solar-124
2021-12-14T14:33:41.219587Z	info	ads	CDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:38 size:33.4kB cached:33/33
2021-12-14T14:33:41.219839Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:33 size:5.8kB empty:0 cached:33/33
2021-12-14T14:33:41.222228Z	info	ads	LDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:32 size:97.6kB
2021-12-14T14:33:41.222805Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T14:47:07.294160Z	info	ads	CDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:35 size:30.6kB cached:30/30
2021-12-14T14:47:07.294403Z	info	ads	EDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:33 size:5.1kB empty:3 cached:30/33
2021-12-14T14:47:07.298203Z	info	ads	LDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:29 size:90.6kB
2021-12-14T14:47:07.299103Z	info	ads	RDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:21 size:11.6kB cached:21/21
2021-12-14T14:47:07.324300Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:30 size:4.9kB empty:0 cached:30/30
2021-12-14T14:47:52.304717Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:20 size:11.6kB cached:20/20
2021-12-14T14:47:52.307537Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:19 size:11.5kB cached:19/19
2021-12-14T14:58:26.053134Z	info	ads	CDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:38 size:33.2kB cached:30/33
2021-12-14T14:58:26.053602Z	info	ads	EDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:30 size:4.9kB empty:0 cached:30/30
2021-12-14T14:58:26.057893Z	info	ads	LDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:32 size:97.6kB
2021-12-14T14:58:26.058280Z	info	ads	RDS: PUSH for node:registry-8644dd67c8-g7b28.solar resources:19 size:11.5kB cached:19/19
2021-12-14T14:58:26.068687Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:33 size:6.0kB empty:0 cached:33/33
2021-12-14T14:58:26.078815Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T15:04:36.550426Z	info	ads	ADS: "10.42.0.156:47564" registry-8644dd67c8-g7b28.solar-124 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T15:04:36.700604Z	info	ads	ADS: new connection for node:registry-8644dd67c8-g7b28.solar-140
2021-12-14T15:04:36.700998Z	info	ads	CDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:38 size:33.2kB cached:32/33
2021-12-14T15:04:36.701191Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:33 size:6.8kB empty:0 cached:33/33
2021-12-14T15:04:36.703357Z	info	ads	LDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:32 size:97.6kB
2021-12-14T15:04:36.703594Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T15:32:22.599914Z	info	ads	ADS: "10.42.0.156:47078" registry-8644dd67c8-g7b28.solar-140 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T15:32:22.820841Z	info	ads	ADS: new connection for node:registry-8644dd67c8-g7b28.solar-157
2021-12-14T15:32:22.821275Z	info	ads	CDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:38 size:33.2kB cached:32/33
2021-12-14T15:32:22.821424Z	info	ads	EDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:33 size:6.8kB empty:0 cached:33/33
2021-12-14T15:32:22.823469Z	info	ads	LDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:32 size:97.6kB
2021-12-14T15:32:22.823819Z	info	ads	RDS: PUSH request for node:registry-8644dd67c8-g7b28.solar resources:21 size:12.8kB cached:21/21
2021-12-14T15:53:59.492346Z	info	Sidecar injection request for solar/registry-5b548db5-***** (actual name not yet known)
2021-12-14T15:54:02.801102Z	info	ads	Push debounce stable[184] 1 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.332496ms since last change, 100.332092ms since last push, full=false
2021-12-14T15:54:09.768959Z	info	ads	ADS: new connection for node:registry-5b548db5-mm89t.solar-163
2021-12-14T15:54:09.769382Z	info	ads	CDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:38 size:33.2kB cached:32/33
2021-12-14T15:54:09.794525Z	info	ads	EDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:33 size:6.8kB empty:0 cached:33/33
2021-12-14T15:54:10.058341Z	info	ads	LDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:32 size:97.6kB
2021-12-14T15:54:10.116561Z	info	ads	RDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:21 size:12.8kB cached:21/21
2021-12-14T15:54:11.658340Z	info	ads	CDS: PUSH for node:registry-5b548db5-mm89t.solar resources:38 size:33.2kB cached:33/33
2021-12-14T15:54:11.658682Z	info	ads	EDS: PUSH for node:registry-5b548db5-mm89t.solar resources:33 size:6.8kB empty:0 cached:33/33
2021-12-14T15:54:11.662863Z	info	ads	LDS: PUSH for node:registry-5b548db5-mm89t.solar resources:32 size:97.6kB
2021-12-14T15:54:11.663360Z	info	ads	RDS: PUSH for node:registry-5b548db5-mm89t.solar resources:21 size:12.8kB cached:21/21
2021-12-14T15:54:11.807639Z	info	ads	Push debounce stable[185] 2 for config ServiceEntry/solar/registry.solar.svc.cluster.local: 100.60596ms since last change, 129.362352ms since last push, full=false
2021-12-14T15:54:16.748195Z	info	ads	ADS: "10.42.0.156:47948" registry-8644dd67c8-g7b28.solar-157 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T16:06:55.903325Z	info	ads	CDS: PUSH for node:registry-5b548db5-mm89t.solar resources:38 size:33.4kB cached:31/33
2021-12-14T16:06:55.907862Z	info	ads	EDS: PUSH for node:registry-5b548db5-mm89t.solar resources:33 size:6.9kB empty:0 cached:33/33
2021-12-14T16:06:55.911308Z	info	ads	LDS: PUSH for node:registry-5b548db5-mm89t.solar resources:32 size:97.6kB
2021-12-14T16:06:55.912052Z	info	ads	RDS: PUSH for node:registry-5b548db5-mm89t.solar resources:21 size:12.8kB cached:21/21
2021-12-14T16:26:42.229371Z	info	ads	ADS: "10.42.0.211:54934" registry-5b548db5-mm89t.solar-163 terminated rpc error: code = Canceled desc = context canceled
2021-12-14T16:26:42.673339Z	info	ads	ADS: new connection for node:registry-5b548db5-mm89t.solar-172
2021-12-14T16:26:42.674026Z	info	ads	CDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:38 size:33.4kB cached:33/33
2021-12-14T16:26:42.674325Z	info	ads	EDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:33 size:6.9kB empty:0 cached:33/33
2021-12-14T16:26:42.678303Z	info	ads	LDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:32 size:97.6kB
2021-12-14T16:26:42.678610Z	info	ads	RDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:21 size:12.8kB cached:21/21
2021-12-14T16:28:22.815684Z	info	ads	CDS: PUSH for node:registry-5b548db5-mm89t.solar resources:38 size:33.4kB cached:33/33
2021-12-14T16:28:22.815865Z	info	ads	EDS: PUSH for node:registry-5b548db5-mm89t.solar resources:33 size:6.8kB empty:0 cached:33/33
2021-12-14T16:28:22.819245Z	info	ads	LDS: PUSH for node:registry-5b548db5-mm89t.solar resources:32 size:97.6kB
2021-12-14T16:28:22.822572Z	info	ads	RDS: PUSH for node:registry-5b548db5-mm89t.solar resources:21 size:12.8kB cached:21/21
2021-12-14T16:38:02.092743Z	info	ads	CDS: PUSH for node:registry-5b548db5-mm89t.solar resources:35 size:30.6kB cached:30/30
2021-12-14T16:38:02.093178Z	info	ads	EDS: PUSH for node:registry-5b548db5-mm89t.solar resources:33 size:5.1kB empty:3 cached:30/33
2021-12-14T16:38:02.098049Z	info	ads	LDS: PUSH for node:registry-5b548db5-mm89t.solar resources:29 size:90.6kB
2021-12-14T16:38:02.099095Z	info	ads	RDS: PUSH for node:registry-5b548db5-mm89t.solar resources:21 size:11.6kB cached:21/21
2021-12-14T16:38:02.118082Z	info	ads	EDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:30 size:4.9kB empty:0 cached:30/30
2021-12-14T16:38:47.102897Z	info	ads	RDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:20 size:11.6kB cached:20/20
2021-12-14T16:38:47.105583Z	info	ads	RDS: PUSH request for node:registry-5b548db5-mm89t.solar resources:19 size:11.5kB cached:19/19

```
</details>

Pull request:
Only get hostnames at endpoint removal.
Fix #36510.

When an endpoint slice gets removed, the corresponding service could also be removed, and query service registry for the service for that service would not return anything and some stale endpoint will be kept in the cache. This change make removal logic go through all possible host names for a slice removal.

Patch:
diff --git a/istio.deps b/istio.deps
index f2a17754b5..ea0577b594 100644
--- a/istio.deps
+++ b/istio.deps
@@ -4,6 +4,6 @@
     "name": "PROXY_REPO_SHA",
     "repoName": "proxy",
     "file": "",
-    "lastStableSHA": "58bb5bd4bb559c31f488c3ea489b76c634d2e427"
+    "lastStableSHA": "e0dfbac609dbcafd1e9024a10083276508d92b67"
   }
 ]
diff --git a/pilot/pkg/serviceregistry/kube/controller/endpointslice.go b/pilot/pkg/serviceregistry/kube/controller/endpointslice.go
index f563a9eab0..b7ce95e794 100644
--- a/pilot/pkg/serviceregistry/kube/controller/endpointslice.go
+++ b/pilot/pkg/serviceregistry/kube/controller/endpointslice.go
@@ -202,11 +202,12 @@ func (esc *endpointSliceController) forgetEndpoint(endpoint interface{}) map[hos
 	}
 
 	out := make(map[host.Name][]*model.IstioEndpoint)
-	for _, svc := range esc.c.servicesForNamespacedName(esc.getServiceNamespacedName(slice)) {
+	for _, hostName := range esc.c.hostNamesForNamespacedName(esc.getServiceNamespacedName(slice)) {
 		// endpointSlice cache update
-		hostName := svc.Hostname
-		esc.endpointCache.Delete(hostName, slice.Name)
-		out[hostName] = esc.endpointCache.Get(hostName)
+		if esc.endpointCache.Has(hostName) {
+			esc.endpointCache.Delete(hostName, slice.Name)
+			out[hostName] = esc.endpointCache.Get(hostName)
+		}
 	}
 	return out
 }
@@ -438,6 +439,13 @@ func (e *endpointSliceCache) Get(hostname host.Name) []*model.IstioEndpoint {
 	return endpoints
 }
 
+func (e *endpointSliceCache) Has(hostname host.Name) bool {
+	e.mu.RLock()
+	defer e.mu.RUnlock()
+	_, found := e.endpointsByServiceAndSlice[hostname]
+	return found
+}
+
 func endpointSliceSelectorForService(name string) klabels.Selector {
 	return klabels.Set(map[string]string{
 		v1beta1.LabelServiceName: name,
diff --git a/pilot/pkg/xds/endpoint_builder.go b/pilot/pkg/xds/endpoint_builder.go
index 2dbb3a6a10..6b061ca646 100644
--- a/pilot/pkg/xds/endpoint_builder.go
+++ b/pilot/pkg/xds/endpoint_builder.go
@@ -274,7 +274,7 @@ func (b *EndpointBuilder) buildLocalityLbEndpointsFromShards(
 	// and should, therefore, not be accessed from outside the cluster.
 	isClusterLocal := b.clusterLocal
 
-	shards.mutex.RLock()
+	shards.mutex.Lock()
 	// Extract shard keys so we can iterate in order. This ensures a stable EDS output. Since
 	// len(shards) ~= number of remote clusters which isn't too large, doing this sort shouldn't be
 	// too problematic. If it becomes an issue we can cache it in the EndpointShards struct.
@@ -339,7 +339,7 @@ func (b *EndpointBuilder) buildLocalityLbEndpointsFromShards(
 			locLbEps.append(ep, ep.EnvoyEndpoint, ep.TunnelAbility)
 		}
 	}
-	shards.mutex.RUnlock()
+	shards.mutex.Unlock()
 
 	locEps := make([]*LocLbEndpointsAndOptions, 0, len(localityEpMap))
 	locs := make([]string, 0, len(localityEpMap))
diff --git a/pkg/config/analysis/analyzers/util/config.go b/pkg/config/analysis/analyzers/util/config.go
index ea73c8e95f..6d69196149 100644
--- a/pkg/config/analysis/analyzers/util/config.go
+++ b/pkg/config/analysis/analyzers/util/config.go
@@ -37,16 +37,6 @@ func IsIstioControlPlane(r *resource.Instance) bool {
 	return false
 }
 
-// IsIncluded check if the term exists in a slice of string
-func IsIncluded(slice []string, term string) bool {
-	for _, val := range slice {
-		if val == term {
-			return true
-		}
-	}
-	return false
-}
-
 // IsMatched check if the term can be matched in a slice of string
 func IsMatched(slice []string, term string) bool {
 	for _, val := range slice {
diff --git a/releasenotes/notes/36510.yaml b/releasenotes/notes/36510.yaml
new file mode 100644
index 0000000000..a72d16da63
--- /dev/null
+++ b/releasenotes/notes/36510.yaml
@@ -0,0 +1,8 @@
+apiVersion: release-notes/v2
+kind: bug-fix
+area: traffic-management
+issue:
+  - https://github.com/istio/istio/issues/36510
+releaseNotes:
+  - |
+    **Fixed** an issue where stale endpoints can be configured when a service gets deleted and created again.


Test patch:
diff --git a/pilot/pkg/serviceregistry/kube/controller/endpointslice_test.go b/pilot/pkg/serviceregistry/kube/controller/endpointslice_test.go
index 3b687456ca..fbb7350273 100644
--- a/pilot/pkg/serviceregistry/kube/controller/endpointslice_test.go
+++ b/pilot/pkg/serviceregistry/kube/controller/endpointslice_test.go
@@ -135,7 +135,9 @@ func TestEndpointSliceCache(t *testing.T) {
 	if !testEndpointsEqual(cache.Get(hostname), []*model.IstioEndpoint{ep1}) {
 		t.Fatalf("unexpected endpoints")
 	}
-
+	if !cache.Has(hostname) {
+		t.Fatalf("expect to find the host name")
+	}
 	// add a new endpoint
 	ep2 := &model.IstioEndpoint{
 		Address:         "2.3.4.5",