Issue 1:
Kubernetes version (use kubectl version):

1.3.4

What happened:

Performing a kubectl get jobs -l <any label selector> returns no jobs despite their existence.

What you expected to happen:

List of jobs matching the label selector

How to reproduce it (as minimally and precisely as possible):

Create a job with a label. Call kubectl get jobs -l <selector>

Anything else do we need to know:

I am not sure if this is similar to #22517 or not

Issue 2:
Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.): no

What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): conntrack

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT

Kubernetes version (use kubectl version): tested on the following:

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:13:23Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.5", GitCommit:"b0deb2eb8f4037421077f77cb163dbb4c0a2a9f5", GitTreeState:"clean", BuildDate:"1970-01-01T00:00:00Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Client Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.6", GitCommit:"ae4550cc9c89a593bcda6678df201db1b208133b", GitTreeState:"clean", BuildDate:"2016-08-26T18:13:23Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"3", GitVersion:"v1.3.0", GitCommit:"283137936a498aed572ee22af6774b6fb6e9fd94", GitTreeState:"clean", BuildDate:"2016-07-01T19:19:19Z", GoVersion:"go1.6.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

Cloud provider or hardware configuration: tested on minikube and an AWS cluster

OS (e.g. from /etc/os-release): tested on boot2docker and debian jessie

Kernel (e.g. uname -a): tested on Linux 4.4.15-boot2docker #1 SMP Fri Jul 15 18:11:45 UTC 2016 x86_64 GNU/Linux and Linux 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-2+deb8u3 (2016-07-02) x86_64 GNU/Linux

Install tools: minikube and custom

Others:

What happened:

When attempting to connect to a service which transiently has no running pods, the connect operation hangs, and continues to hang even once brand new connections are able to connect.

What you expected to happen:

The connect operation should hang until a pod is running, then succeed

How to reproduce it (as minimally and precisely as possible):

create the foo http service with no pods (for now)

$ cat > svc.yaml <<EOF
apiVersion: v1
kind: Service
metadata:
name: foo
spec:
ports:
- port: 80
selector:
name: foo
EOF
$ kubectl create -f svc.yaml

start a pod (just to be a shell inside the cluster), and install curl

$ kubectl run -it bar --image=ubuntu

bar# apt-get update && apt-get install curl

attempt to curl the foo service, it should hang trying to connect (this is expected)

bar# curl -v foo.default.svc.cluster.local/

(while the curl is still hanging, before the connect operation times out)

create the foo deployment as a basic nginx pod

$ cat > dep.yaml <<EOF
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: foo
spec:
replicas: 1
template:
metadata:
labels:
name: foo
spec:
containers:
- name: foo
image: nginx
ports:
- containerPort: 80
EOF
$ kubectl create -f dep.yaml

EXPECTED:

Since the service is now available, the connect should stop hanging and instead succeed.

ACTUAL:

The connect continues to hang. If you abort and retry with a brand new connection, it succeeds immediately.

Anything else do we need to know:

The above is a minimal case, but I originally hit this issue when testing rolling upgrades of a deployment with 1 replica. Since there's a brief window where there are no running pods for the service, there is a race where a connection can enter this state.

I have even observed this happening in cases where only one replica is running at some stage, eg. when a deployment with default strategy has replicas=2. However, this is much harder to trigger, suggesting a much smaller race window.

I think I already know what's actually happening - this is the same bug as #31983 but applied to TCP, not UDP. See also moby/moby#8795.

When the opening SYN packet for the TCP connection arrives, it gets passed to an empty service routing chain, and subsequently is not rewritten. In my tests, this caused it to go out the default route instead (ie. out of the minikube vm and into the host OS), but that may depend on your exact networking setup. In any case the effect is the same - the SYN packet has nowhere to go, and either gets dropped or goes somewhere nonsensical (that's probably a seperate bug, come to think). The issue comes when we add a pod to that service routing chain, and now any new connections will follow that rule and successfully connect.

However, since our first SYN packet went somewhere else, conntrack ensures that it continues to be sent the same place for the duration of that particular TCP connection (even though the "connection" has been nothing but outgoing SYNs so far). So all subsequent SYN retries also end up dropped or otherwise ignored.

I confirmed this was the issue by running conntrack -F on the minikube host - this allowed the hung connection to successfully connect to the newly running pod.

I believe the proper fix for this would be something like "when a service goes from having no pods to having at least one, clear all conntrack entries to that service ip in state SYN_SENT".