Issue 1:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.8", GitCommit:"3040f87c570a772ce94349b379f41f329494a4f7", GitTreeState:"clean", BuildDate:"2016-09-18T21:06:37Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"4+", GitVersion:"v1.4.0-beta.8", GitCommit:"3040f87c570a772ce94349b379f41f329494a4f7", GitTreeState:"clean", BuildDate:"2016-09-18T21:00:36Z", GoVersion:"go1.6.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: GCE+CoreOS+custom k8s

OS (e.g. from /etc/os-release): CoreOS

Kernel (e.g. uname -a): 4.6.3-coreos

Install tools:

Others:

What happened:

Could not access service via external LB from a kubernetes pod on the node that does not host a pod for that service.

What you expected to happen:

Traffic should have gone to the external LB which would pick the node that is running a pod for that service.

TL;DR External load balancer kube-proxy iptable rules should only match on traffic originating from outside the cluster/pod networks i.e. from the external load balancer.

Otherwise traffic originating from within k8s cluster to external load balanced ip addresses can get blackholed.

We are having an other issue with OnlyLocal.

I have a service with the following definition

{
"apiVersion": "v1",
"kind": "Service",
"metadata": {
"annotations": {
"service.alpha.kubernetes.io/external-traffic": "OnlyLocal"
},
"labels": {
"k8s-app": "my-app",
"kubernetes.io/cluster-service": "true"
},
"name": "my-app",
"namespace": "default"
},
"spec": {
"loadBalancerIP": "1.2.3.4",
"ports": [
{
"name": "http",
"port": 80,
"targetPort": 80
}
],
"selector": {
"k8s-app": "my-app"
},
"type": "LoadBalancer"
}
}

If I run

curl http://1.2.3.4/test

From my local box that is outside of the k8s cluster I get a response.

When I run the same command from a k8s node that is not the node that is hosting the pod for my-app I get no response.

curl http://1.2.3.4/test -v
* Trying 1.2.3.4

Checking the iptables rules. Some stuff truncated:

-A KUBE-SERVICES -d 1.2.3.4/32 -p tcp -m comment --comment "default/my-app:http loadbalancer IP" -m tcp --dport 80 -j KUBE-FW-ABZR2FBH2NLOJM2K
-A KUBE-FW-ABZR2FBH2NLOJM2K -m comment --comment "default/my-app:http loadbalancer IP" -j KUBE-XLB-ABZR2FBH2NLOJM2K
-A KUBE-XLB-ABZR2FBH2NLOJM2K -m comment --comment "default/my-app:http has no local endpoints" -j KUBE-MARK-DROP

So it looks like the traffic gets dropped by the kube-proxy.

The KUBE-SERVICES seems to match packets that originate from pods on this machine and not only external traffic. This means that if a pod on this machine wants to call a service through the external LB

(e.g. when using external DNS registrations of services) the traffic will be blackholed if the given node does not have a pod in question.

Proposed solution would be to augment the KUBE-SERVICES rule to filter out traffic originating from clusters/pods private network.

This is related to #29409

@girishkalele

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.):

No

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.):

watch leak

Is this a BUG REPORT or FEATURE REQUEST? (choose one):

BUG REPORT

Kubernetes version (use kubectl version):

1.4.0

Environment:

Cloud provider or hardware configuration: vagrant on Mac OS X

OS (e.g. from /etc/os-release): CoreOS 1164.1.0

Kernel (e.g. uname -a): Linux core-01 4.7.3-coreos #1 SMP Sat Sep 10 08:21:12 UTC 2016 x86_64 Intel(R) Core(TM) i7-4980HQ CPU @ 2.80GHz GenuineIntel GNU/Linux

Install tools:

Others:

What happened:

We ran controller which handles our own ThirdPartyResource.

When we hit "watch of ... ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared" error, "forcing resync" log is duplicated.

This log message comes from here. It looks like this goroutine does not exit when we hit the above mentioned error, and continued to work, and running "resync" more frequently.

What you expected to happen:

The resync goroutine must be terminated when watch is closed (i.e., ListAndWatch function returned), instead of leaking it.

How to reproduce it (as minimally and precisely as possible):

I just run our controller for ~15 minutes. I'm not sure why "401: The event in requested index is outdated and cleared (the requested history has been cleared" happens, but from the issues I read, it should be harmless.

Anything else do we need to know: