Repository: Azure/AKS

Issue #732: kube-svc-redirect is incredibly noisy
Created: 2018-11-23T17:46:39Z
What happened:
We get a ton of logging output from the redirector constantly

Nov 23 09:43:14 ec2-18-222-113-148.us-east-2.compute.amazonaws.com instadata-sqs-scan DEBUG BEGIN SCAN
Nov 23 09:43:42 exchange-99555c4ff-4w4zt exchange DEBUG +0000] [10] [DEBUG] GET /
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  Chain aks-hcp-custom-svc (2 references)
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  target     prot opt source               destination         
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  DNAT       tcp  --  anywhere             10.0.0.1             to:127.0.0.1:14612
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  RETURN     all  --  anywhere             anywhere            
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  [ 17:43:47 ] INF: found expected rule count in chain:aks-hcp-custom-svc
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  [ 17:43:47 ] INF: Will validate the following rules:
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  +++++++++++++++++++++++++++++++++++
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  Chain aks-hcp-custom-svc (2 references)
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  num  target     prot opt source               destination         
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  1    DNAT       tcp  --  anywhere             10.0.0.1             to:127.0.0.1:14612
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  2    RETURN     all  --  anywhere             anywhere            
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  +++++++++++++++++++++++++++++++++++
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  [ 17:43:47 ] INF: Rule #1 match expectation with [1    DNAT       tcp  --  anywhere             10.0.0.1             to:127.0.0.1:14612]
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  [ 17:43:47 ] INF: Rule #2 match expectation with [2    RETURN     all  --  anywhere             anywhere            ]
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  [ 17:43:47 ] INF: chain aks-hcp-custom-svc is valid
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  [ 17:43:47 ] INF: Jump rule aks-hcp-custom-svc was found at position #1 in OUTPUT
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  [ 17:43:47 ] INF: Rules in OUTPUT are:
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  +++++++++++++++++++++++++++++++++++
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  Chain OUTPUT (policy ACCEPT)
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  num  target     prot opt source               destination         
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  1    aks-hcp-custom-svc  all  --  anywhere             anywhere            
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  2    KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  3    DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  +++++++++++++++++++++++++++++++++++
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  [ 17:43:47 ] INF: Jump rule aks-hcp-custom-svc was found at position #1 in PREROUTING
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  [ 17:43:47 ] INF: Rules in PREROUTING are:
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  +++++++++++++++++++++++++++++++++++
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  Chain PREROUTING (policy ACCEPT)
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  num  target     prot opt source               destination         
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  1    aks-hcp-custom-svc  all  --  anywhere             anywhere            
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  2    KUBE-SERVICES  all  --  anywhere             anywhere             /* kubernetes service portals */
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  3    DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  +++++++++++++++++++++++++++++++++++
Nov 23 09:43:47 kube-svc-redirect-dnrtl redirector  [ 17:43:47 ] INF: Done - going to sleep for 10
Nov 23 09:43:52 kube-svc-redirect-p75qc azureproxy  10.244.1.6 - - [23/Nov/2018:17:43:52 +0000] "POST /apis/authorization.k8s.io/v1beta1/subjectaccessreviews HTTP/2.0" 201 406 "-" "metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format" "-" rt=0.008 uct="0.008" uht="0.012" urt="0.012"
Nov 23 09:43:53 kube-svc-redirect-p75qc azureproxy  10.244.1.6 - - [23/Nov/2018:17:43:52 +0000] "POST /apis/authorization.k8s.io/v1beta1/subjectaccessreviews HTTP/2.0" 201 408 "-" "metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format" "-" rt=0.030 uct="0.008" uht="0.032" urt="0.032"
Nov 23 09:43:52 kube-svc-redirect-zlf7b azureproxy  10.244.3.39 - - [23/Nov/2018:17:43:52 +0000] "GET /api/v1/namespaces/honeycomb-system/pods/kube-state-metrics-678654f89f-xdcdn HTTP/1.1" 200 1700 "-" "pod_nanny/v0.0.0 (linux/amd64) kubernetes/$Format" "-" rt=0.058 uct="0.028" uht="0.060" urt="0.060"
Nov 23 09:43:52 kube-svc-redirect-zlf7b azureproxy  10.244.3.25 - - [23/Nov/2018:17:43:52 +0000] "GET /api/v1/namespaces/kube-system/configmaps/cert-manager-controller HTTP/2.0" 200 380 "-" "cert-manager/v0.0.0 (linux/amd64) kubernetes/$Format/leader-election" "-" rt=0.012 uct="0.000" uht="0.008" urt="0.008"

What you expected to happen:
Logging should log interesting thing, not just spam
How to reproduce it (as minimally and precisely as possible):
Just running the cluster.
Anything else we need to know?:
Environment:
Newest AKS cluster

Size of cluster (how many worker nodes are in the cluster?)
2 3 node clusters

Duplicate issue #851: kube-svc-redirect generates huge amount of logs
Created: 2019-02-25T12:58:35Z
What happened*:
kube-svc-redirect generates huge amount of logs, EFK stack is just getting filled up with these logs... any fix for this?
What you expected to happen*:
Only logs whenever acessed
How to reproduce it (as minimally and precisely as possible):
Check pod log in kube-system
Anything else we need to know?:
Environment:

AKS 2 node, kubenetes v 1.12

Closing comment (2019-04-02T20:51:47Z): Duplicate of #732