Change:
The symptom is the nodeport service is accessible thru hostip:hostport. But the port on host is not occupied. This is exposed by e2e on kubenet. I noticed in the kube- Kube-proxy did open the port. But later, kube-proxy closed the ports upon iptables-restore failure. It never reopen the ports. Hence causing the e2e failure. The code section in question: Not sure why failed. Error messge is not helpful since line 139 is ref:
Also of interest: did it exit with 1? This should be clear from the iptables commands it executed before the commit, which are hopefully in the logs (can you post them). If it's something like a non-exist chain, we should fix that. we retry on exit 1? Handwaving without looking at code: I'd expect something to get requeued, and to reinvoke syncProxyRules once again, in a time frame << test timeout.
Yeah, what was line 139 in the restore output? , Prashanth B wrote:
is No retry though. It is going to catch up in the next sync. So I think no retry for is needed.
FYI Jenkins job:
That iptables restore is 139 lines long, so it's unhappy about the last line? can you hunt ths? , Minhan Xia wrote:
The next sync is whenver the next service/endpoint changes right? Errors during the sync should cause an immediate retry of sync, preferably with backoff. If a service or endpoint never changes we should still recover. Yes we should hunt down the bad line, but it's not the last one right? COMMIT will fail if eg: Even though it's obviously not commit but the tcp/upd thing.
iptables resyncs every 30s, not ideal but in this case I don't expect that to fix the error I meant that we need to root cause it :) , Prashanth B wrote:
But in this case, iptables-restore is going to complain . Not line 2
Hmm
I manually restored the 139 lines above it worked fine - no errors. Even more curious. , Prashanth B wrote:
I also tried to execute the same sequence of iptables-restore. It worked. No errors.
Was this a one-time thing? Were there any other changes in your git repo? , Minhan Xia wrote:
This happened in kubenet suite. Only difference is . I looked at a few runs where e2e failed. This happened 3-4 times over the whole test, but not consecutively. In all-passed runs I checked, this never happened (Maybe I am lucky).
Of , Minhan Xia wrote:

Discussion:
// Accumulate NAT chains to keep. activeNATChains := map[utiliptables.Chain]bool{} // use a map as a set <del> // Accumulate new local ports that we have opened. newLocalPorts := map[localPort]closeable{} </del> <ins> // Accumulate the set of local ports that we will be holding open once this update is complete replacementPortsMap := map[localPort]closeable{} </ins> // Build rules for each service. for svcName, svcInfo := range proxier.serviceMap {

protocol: protocol, } if proxier.portsMap[lp] != nil { <del> newLocalPorts[lp] = proxier.portsMap[lp] </del> <ins> glog.V(4).Infof("Port %s was open before and is still needed", lp.String()) replacementPortsMap[lp] = proxier.portsMap[lp] </ins> } else { socket, err := openLocalPort(&lp) if err != nil { glog.Errorf("can't open %s, skipping this externalIP: %v", lp.String(), err) continue } <del> newLocalPorts[lp] = socket </del> <ins> replacementPortsMap[lp] = socket </ins> } } // We're holding the port, so it's OK to install iptables rules. args := []string{

protocol: protocol, } if proxier.portsMap[lp] != nil { <del> newLocalPorts[lp] = proxier.portsMap[lp] </del> <ins> glog.V(4).Infof("Port %s was open before and is still needed", lp.String()) replacementPortsMap[lp] = proxier.portsMap[lp] </ins> } else { socket, err := openLocalPort(&lp) if err != nil { glog.Errorf("can't open %s, skipping this nodePort: %v", lp.String(), err) continue } <del> newLocalPorts[lp] = socket </del> <ins> replacementPortsMap[lp] = socket </ins> } // We're holding the port, so it's OK to install iptables rules. args := []string{

if err != nil { glog.Errorf("Failed to execute iptables-restore: %v", err) // Revert new local ports. <del> for k, v := range newLocalPorts { glog.Errorf("Closing local port %s", k.String()) v.Close() } </del> <ins> revertPorts(replacementPortsMap, proxier.portsMap) </ins> return } // Close old local ports and save new ones. for k, v := range proxier.portsMap { <del> if newLocalPorts[k] == nil { </del> <ins> if replacementPortsMap[k] == nil { </ins> v.Close() } } <del> proxier.portsMap = newLocalPorts </del> <ins> proxier.portsMap = replacementPortsMap </ins> // Clean up the older SNAT rule which was directly in POSTROUTING. // TODO(thockin): Remove this for v1.3 or v1.4.

glog.V(2).Infof("Opened local port %s", lp.String()) return socket, nil } <ins> // revertPorts is closing ports in replacementPortsMap but not in originalPortsMap. In other words, it only // closes the ports opened in this sync. func revertPorts(replacementPortsMap, originalPortsMap map[localPort]closeable) { for k, v := range replacementPortsMap { // Only close newly opened local ports - leave ones that were open before this update if originalPortsMap[k] == nil { glog.V(2).Infof("Closing local port %s after iptables-restore failure", k.String()) v.Close() } } } </ins>

} } <ins> type fakeClosable struct { closed bool } func (c *fakeClosable) Close() error { c.closed = true return nil } func TestRevertPorts(t *testing.T) { testCases := []struct { replacementPorts []localPort existingPorts    []localPort expectToBeClose  []bool }{ { replacementPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, existingPorts:   []localPort{}, expectToBeClose: []bool{true, true, true}, }, { replacementPorts: []localPort{}, existingPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, expectToBeClose: []bool{}, }, { replacementPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, existingPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, expectToBeClose: []bool{false, false, false}, }, { replacementPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, existingPorts: []localPort{ {port: 5001}, {port: 5003}, }, expectToBeClose: []bool{false, true, false}, }, { replacementPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, }, existingPorts: []localPort{ {port: 5001}, {port: 5002}, {port: 5003}, {port: 5004}, }, expectToBeClose: []bool{false, false, false}, }, } for i, tc := range testCases { replacementPortsMap := make(map[localPort]closeable) for _, lp := range tc.replacementPorts { replacementPortsMap[lp] = &fakeClosable{} } existingPortsMap := make(map[localPort]closeable) for _, lp := range tc.existingPorts { existingPortsMap[lp] = &fakeClosable{} } revertPorts(replacementPortsMap, existingPortsMap) for j, expectation := range tc.expectToBeClose { if replacementPortsMap[tc.replacementPorts[j]].(*fakeClosable).closed != expectation { t.Errorf("Expect replacement localport %v to be %v in test case %v", tc.replacementPorts[j], expectation, i) } } for _, lp := range tc.existingPorts { if existingPortsMap[lp].(*fakeClosable).closed == true { t.Errorf("Expect existing localport %v to be false in test case %v", lp, i) } } } } </ins> // TODO(thockin): add a test for syncProxyRules() or break it down further and test the pieces.