Issue 1:
Hello all - apologies if I'm creating a duplicate ticket...I haven't been able to conjure exactly what I'm looking for; this may be related to the issues described here: https://groups.google.com/forum/#!topic/google-containers/yg539XUMQSA. I have tried going through the services walkthrough, to no avail.

I am creating authenticated Mongo replica sets using Kubernetes on GCE. We are using custom, minimal images to run our software - I have tried the official images but have had the same problems with those.

I can get Mongo to run without a hitch - the pods talk to each other, failed primaries trigger an election, syncing of documents occurs just fine.

My troubles come in the form of the load balanced IP address assigned to these instances. Connections to the pods via the LB address are very unreliable, and recently have ceased to function at all. Connecting to the whole replica set is impossible.

Our implementation does the following:

creates a persistent disk for the instance if it does not exist (gcloud compute disks create ...),

creates a PV/PVC binding to the disk,

creates a Service for each instance (with type: LoadBalancer),

creates a replication controller for each Mongo instance.

We're having to make a lot of these, so I'm using a pidgin template to generate all of these at runtime. In any event, all pods and services come up, with each service binding to the correct pod. Case in point:

$ kubectl get pods | grep api-mongo
api-mongo-1-w358n 1/1 Running 0 1m
api-mongo-2-6j048 1/1 Running 0 59s
api-mongo-3-f8p0j 1/1 Running 0 57s

$ kubectl get svc | grep api-mongo
api-mongo-1 10.0.184.8 104.196.2.106 41572/TCP name=api-mongo-1,tier=data,track=stable 1m
api-mongo-2 10.0.27.185 104.196.36.137 41572/TCP name=api-mongo-2,tier=data,track=stable 1m
api-mongo-3 10.0.63.141 104.196.30.9 41572/TCP name=api-mongo-3,tier=data,track=stable 1m

$ kubectl get pods --selector name=api-mongo-1,tier=data,track=stable
NAME READY STATUS RESTARTS AGE
api-mongo-1-w358n 1/1 Running 0 2m

$ kubectl get pods api-mongo-1-w358n -o json | grep podIP
"podIP": "10.244.6.8",

$ kubectl get endpoints api-mongo-1
NAME ENDPOINTS AGE
api-mongo-1 10.244.6.8:41572 4m

This is as I would expect. When I attempt to shell into to Mongo, however:

$ mongo -u admin --host 104.196.2.106 -p **** --port 41572 --authenticationDatabase admin admin
MongoDB shell version: 3.2.0
connecting to: 104.196.2.106:41572/admin
2016-02-05T10:47:57.113-0500 W NETWORK [thread1] Failed to connect to 104.196.2.106:41572 after 5000 milliseconds, giving up.
2016-02-05T10:47:57.113-0500 E QUERY [thread1] Error: couldn't connect to server 104.196.2.106:41572, connection attempt failed :
connect@src/mongo/shell/mongo.js:226:14
@(connect):1:6

exception: connect failed

Yesterday, I was able to shell in every 5-6 attempts - today, it never happens.

Here's how the configurations look when rendered:

Service:

kind: "Service"
metadata:
name: "api-mongo-1"
labels:
name: "api-mongo-1"
tier: "data"
track: "stable"
spec:
type: "LoadBalancer"
ports:
- name: "mongodb"
port: 41572
targetPort: 41572
selector:
name: "api-mongo-1"
tier: "data"
track: "stable"

Replication Controller (truncated of environment variables and volumes):

apiVersion: "v1"
kind: "ReplicationController"
metadata:
name: "api-mongo-1"
labels:
name: "api-mongo-1"
tier: "data"
track: "stable"
spec:
replicas: 1
selector:
name: "api-mongo-1"
tier: "data"
track: "stable"
template:
metadata:
labels:
name: "api-mongo-1"
tier: "data"
track: "stable"
spec:
containers:
- name: "api-mongo-1"
image: "mongo:3.2.0"
ports:
- name: "mongo"
containerPort: 41572

iptables-save on the node:

# iptables-save | grep api-mongo-1
-A KUBE-NODEPORT-CONTAINER -p tcp -m comment --comment "default/api-mongo-1:mongodb" -m tcp --dport 32113 -j REDIRECT --to-ports 45347
-A KUBE-NODEPORT-HOST -p tcp -m comment --comment "default/api-mongo-1:mongodb" -m tcp --dport 32113 -j DNAT --to-destination 10.240.0.19:45347
-A KUBE-PORTALS-CONTAINER -d 10.0.184.8/32 -p tcp -m comment --comment "default/api-mongo-1:mongodb" -m tcp --dport 41572 -j REDIRECT --to-ports 45347
-A KUBE-PORTALS-CONTAINER -d 104.196.2.106/32 -p tcp -m comment --comment "default/api-mongo-1:mongodb" -m tcp --dport 41572 -j REDIRECT --to-ports 45347
-A KUBE-PORTALS-HOST -d 10.0.184.8/32 -p tcp -m comment --comment "default/api-mongo-1:mongodb" -m tcp --dport 41572 -j DNAT --to-destination 10.240.0.19:45347
-A KUBE-PORTALS-HOST -d 104.196.2.106/32 -p tcp -m comment --comment "default/api-mongo-1:mongodb" -m tcp --dport 41572 -j DNAT --to-destination 10.240.0.19:45347

Mongo responds correctly to cURL requests from within the node, whether I'm making the request against the LB IP, or directly to the kube proxy on the node.

Does anyone have any ideas about what I'm doing wrong here?

Please let me know if I can provide any further information here. Thanks in advance!

EDIT 1: edited containerPort on the RC - I fired my "dry run" code incorrectly when I posted this, and it rendered a default value in my template.

Issue 2:
The configmap tests should run as part of the node e2e suite. For now the tests can be duplicated in the test/e2e_node directory.

Reserved for @mnshaw