Issue 1:
So running the simplest example of nginx on hyperkube everything goes well. I can reach nginx from other pods in the cluster and from the host. But when I try to curl the service ip from within the pod itself then I get no response.

The iptables rules below illustrate the problem

carsonanderson ~ # iptables-save |egrep "10.0.0.25|KUBE-SVC-4N57TFCL4MD7ZTDA|KUBE-SEP-YAIKK2SEJQZ5ZUOS|MARK"
:KUBE-MARK-MASQ - [0:0]
:KUBE-SEP-YAIKK2SEJQZ5ZUOS - [0:0]
:KUBE-SVC-4N57TFCL4MD7ZTDA - [0:0]
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-SEP-YAIKK2SEJQZ5ZUOS -s 172.17.0.2/32 -m comment --comment "default/nginx:" -j KUBE-MARK-MASQ
-A KUBE-SEP-YAIKK2SEJQZ5ZUOS -p tcp -m comment --comment "default/nginx:" -m tcp -j DNAT --to-destination 172.17.0.2:80
-A KUBE-SERVICES -d 10.0.0.25/32 -p tcp -m comment --comment "default/nginx: cluster IP" -m tcp --dport 80 -j KUBE-SVC-4N57TFCL4MD7ZTDA
-A KUBE-SVC-4N57TFCL4MD7ZTDA -m comment --comment "default/nginx:" -j KUBE-SEP-YAIKK2SEJQZ5ZUOS

I think it's because we are marking it for masquerade and then sending the traffic out of the network when the source is one of the pods serving a service.

The -A KUBE-SEP-YAIKK2SEJQZ5ZUOS -s 172.17.0.2/32 -m comment --comment "default/nginx:" -j KUBE-MARK-MASQ line should probably also take into account if the destination is the same as the source and route back into itself directly or jump to the KUBE-SVC-XXX rule instead of hitting the masquerade table and going out to the world.

Issue 2:
I have a Kubernetes 1.2.2 setup with kube-proxy in iptables mode.

There is a Pod A with Pod-IP 10.2.94.5.

There is a Service A with IP 10.3.249.19, which redirects all traffic to this pod.

There is Pod B with Pod-IP 10.2.94.2.

Pod A and B are on the same machine.

When I access service A from pod B, everything works fine. However, when I try to access service A from pod A, iptables seems to drop the request.

In the examples below, I'm trying to access a squid as a proof-of-concept. The practical use case would be to run a GlusterFS (which needs access to its own outside IP when creating volumes).

I did a bit iptables digging and it seems like there is a masquerading rule KUBE-MARK-MASQ, which catches the request and prevents forwarding it.

Trace for accessing service A from pod B (successful):

[248369.783562] TRACE: raw:PREROUTING:rule:2 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783580] TRACE: raw:PREROUTING:policy:3 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783594] TRACE: nat:PREROUTING:rule:1 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783645] TRACE: nat:KUBE-SERVICES:rule:16 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783670] TRACE: nat:KUBE-SVC-NQAN5OLURYDUMCKE:rule:1 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783676] TRACE: nat:KUBE-SEP-AIYB5BTNB4W7CP2T:rule:2 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783694] TRACE: filter:FORWARD:rule:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783700] TRACE: filter:DOCKER-ISOLATION:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783705] TRACE: filter:FORWARD:rule:2 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783709] TRACE: filter:DOCKER:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783715] TRACE: filter:FORWARD:rule:5 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783720] TRACE: nat:POSTROUTING:rule:1 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783725] TRACE: nat:KUBE-POSTROUTING:return:2 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783729] TRACE: nat:POSTROUTING:rule:2 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783733] TRACE: nat:FLANNEL:rule:1 IN= OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 SRC=10.2.94.2 DST=10.2.94.5 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=47421 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631512 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03B24FA50000000001030307)
[248369.783815] TRACE: raw:PREROUTING:rule:2 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5)
[248369.783824] TRACE: raw:PREROUTING:policy:3 IN=docker0 OUT= PHYSIN=veth384b142 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.3.249.19 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=8080 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5)
[248369.783831] TRACE: filter:FORWARD:rule:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5)
[248369.783836] TRACE: filter:DOCKER-ISOLATION:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5)
[248369.783841] TRACE: filter:FORWARD:rule:2 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5)
[248369.783846] TRACE: filter:DOCKER:return:1 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5)
[248369.783850] TRACE: filter:FORWARD:rule:3 IN=docker0 OUT=docker0 PHYSIN=veth384b142 PHYSOUT=veth5bc2893 MAC=02:42:0a:02:5e:05:02:42:0a:02:5e:02:08:00 SRC=10.2.94.2 DST=10.2.94.5 LEN=52 TOS=0x10 PREC=0x00 TTL=64 ID=47422 DF PROTO=TCP SPT=49802 DPT=3128 SEQ=2455631513 ACK=1223670235 WINDOW=224 RES=0x00 ACK URGP=0 OPT (0101080A03B24FA503B24FA5)

Trace for accessing service A from pod A (unsuccessful):

[246579.045435] TRACE: raw:PREROUTING:rule:2 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307)
[246579.045451] TRACE: raw:PREROUTING:policy:3 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307)
[246579.045460] TRACE: nat:PREROUTING:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307)
[246579.045504] TRACE: nat:KUBE-SERVICES:rule:18 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307)
[246579.045528] TRACE: nat:KUBE-SVC-NQAN5OLURYDUMCKE:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307)
[246579.045534] TRACE: nat:KUBE-SEP-AIYB5BTNB4W7CP2T:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307)
[246579.045548] TRACE: nat:KUBE-MARK-MASQ:rule:1 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307)
[246579.045556] TRACE: nat:KUBE-MARK-MASQ:return:2 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) MARK=0x4000
[246579.045563] TRACE: nat:KUBE-SEP-AIYB5BTNB4W7CP2T:rule:2 IN=docker0 OUT= PHYSIN=veth5bc2893 MAC=02:42:39:b8:3f:9e:02:42:0a:02:5e:05:08:00 SRC=10.2.94.5 DST=10.3.249.19 LEN=60 TOS=0x10 PREC=0x00 TTL=64 ID=21660 DF PROTO=TCP SPT=40434 DPT=8080 SEQ=1293954685 ACK=0 WINDOW=28640 RES=0x00 SYN URGP=0 OPT (020405980402080A03AB7AE10000000001030307) MARK=0x4000
... (repeats this chunk multiple times as it tries connecting again)

The interesting parts from iptables -t nat -L:

Chain PREROUTING (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
DOCKER all -- anywhere anywhere ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-SERVICES all -- anywhere anywhere /* kubernetes service portals */
DOCKER all -- anywhere !loopback/8 ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
KUBE-POSTROUTING all -- anywhere anywhere /* kubernetes postrouting rules */
FLANNEL all -- 10.2.0.0/16 anywhere
MASQUERADE all -- !10.2.0.0/16 10.2.0.0/16

Chain DOCKER (2 references)
target prot opt source destination

Chain FLANNEL (1 references)
target prot opt source destination
ACCEPT all -- anywhere 10.2.0.0/16
MASQUERADE all -- anywhere !base-address.mcast.net/4

Chain KUBE-MARK-MASQ (50 references)
target prot opt source destination
MARK all -- anywhere anywhere MARK or 0x4000

Chain KUBE-NODEPORTS (1 references)
target prot opt source destination

Chain KUBE-POSTROUTING (1 references)
target prot opt source destination
MASQUERADE all -- anywhere anywhere /* kubernetes service traffic requiring SNAT */ mark match 0x4000/0x4000

Chain KUBE-SEP-AIYB5BTNB4W7CP2T (1 references)
target prot opt source destination
KUBE-MARK-MASQ all -- 10.2.94.5 anywhere /* default/squid:http */
DNAT tcp -- anywhere anywhere /* default/squid:http */ tcp to:10.2.94.5:3128

Chain KUBE-SERVICES (2 references)
target prot opt source destination
[...]
KUBE-SVC-NQAN5OLURYDUMCKE tcp -- anywhere 10.3.249.19 /* default/squid:http cluster IP */ tcp dpt:http-alt
[...]
KUBE-NODEPORTS all -- anywhere anywhere /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL

Chain KUBE-SVC-NQAN5OLURYDUMCKE (1 references)
target prot opt source destination
KUBE-SEP-AIYB5BTNB4W7CP2T all -- anywhere anywhere /* default/squid:http */