Code change:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: https://k8s- Which test(s) are failing: sig-network] SCTP [Feature:SCTP] [LinuxOnly] should create a Pod with SCTP HostPort Since when has it been failing: Test [sig-network] SCTP [Feature:SCTP] [LinuxOnly] should create a Pod with SCTP HostPort has failed 3 times since it last passed at commit/. It started failing between commit/ (Tue Jun 09 13:25 PDT) and commit/ (Wed Jun 10 01:26 PDT): Testgrid link: https://k8s- Reason for failure: Anything else we need to know: I'm not sure if we already did it but there was a proposal to change from dockershim to containerd for the GCE jobs
There are no sig labels on this issue. Please add an appropriate label by using one of the following commands: - Please see the for a listing of the SIGs, working groups, and committees available. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign /cc
GitHub didn't allow me to assign the following users: janosi. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign /cc Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
yes, that would break us... the test checks "are we using kubenet", but it didn't occur to me that we would need to deal with kubenet with CRI instead of kubenet with dockershim. It should be easy to make it check for both the dockershim rule and the CNI portmap rule though...
yes, according to the "Changes" link on the dashboard, containerd replaced docker in GCE right before the failures.
Let's see if I can make it :)
So actually the test is pretty generic (it just looks for and ) and it seems like it ought to pass correctly with CNI portmap too. I'm not sure what's failing. You may need to add some debugging...
OK, so as I can see, containerd's CRI implementation just ignores all HostPort definitions that do not have TCP or UDP Protocol value. I.e. containerd will not send HostPort mapping definitions to the CNI if the protocol is not TCP or UDP.
seems this is the reason why sctp was skipped
Yes, exactly. Introduced right after SCTP support was merged in k8s. Just found this in the package: Though not sure it "knows" the right runtime at runtime, because there is no user of it in :)
just curious, why do we skip in containerd and allow it with dockershim?
Hmm, do you mean why do not we skip in case of containerd? The reason is that I did not remember that containerd PR about the SCTP support. Let's see if I can skip it.
Hm, that is not true. My search engine was wrong. After a restart I found usage of that in e2e_node
Has a bug been filed upstream in containerd? Just skipping the tests seems like the wrong approach here.
sorry, I meant why we are not allowing it in containerd, does this feature should work in all container runtimes? CRIO, docker and containerd? if affirmative, we can submit a PR to containerd/cri to allow SCTP, I can do it if you don't have time
Excuse me, I misunderstood your question. Yep, why not. I mean, it could work in all container runtimes. As I understand the containerd implementation is based on a misunderstanding: that the host port mapping loads the SCTP kernel module and that would be bad for those apps that want to use user space SCTP. But as long as there is no SCTP socket created on a node the SCTP kernel module is not loaded.
I can initiate a discussion with containerd about this.
seems this dates back to the original SCTP KEP, seems like it should be a pretty simple fix (?) thanks for filing the bug and for the details there!
CRI-O seems to have the same issue:
Just confirming you want this filter to be gone now. Or do you need the filter to be manageable in some way.. filtered by a config switch?
For TCP and UDP HostPorts, the runtime is expected to do two things: a listening socket on that port so the kernel will think it's in use and not allocate it to anyone else iptables rules (or whatever) to route traffic from the host port to the container port For SCTP HostPorts, the runtime should do the second part (iptables rules), but should not do the first part (listening socket), for two reasons: Some users want to use "userspace SCTP", in which case they will be referring to SCTP at the Kubernetes level but need to keep the kernel module from ever being loaded. (This is discussed more .) If is not loaded when Kubernetes starts, we want to avoid having Kubernetes itself cause it to be auto-loaded, for security reasons. (More discussion of this .) So, runtimes should be filtering out SCTP for the opening-listening-sockets part, but not for the making-iptables-rules part. (Some implementations, notably CNI , don't actually do the opening-listening-sockets part, so they wouldn't be filtering at all.)
Exactly. The current state in the runtimes: Docker (dockershim, actually): OK. The logic was implemented together with the basic SCTP feature, so listening on the host port is filtered for SCTP, but the iptables rules are created Containerd (CRI, actually): the patch which was applied 2 years ago filters out every SCTP host port handling, because it filters SCTP ports from the list of port mappings that are passed to CNI (portmap) later. Actually, containerd (CRI) does not perform the "listen" part at all, so the patch is not needed at all CRI-O: the patch applied 2 years ago filters out everything. CRI-O imports the host port manager implementation from kubelet/dockershim. For this reason it has the logic that can filter out SCTP ports from the "listen" part, but still applies the iptables rules. I.e. the patch is not needed in CRI-O either.
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale The test is flaky now :thinking: https://k8s-
I guess it is a timing issue? Do you think some extra polling rounds would help? (I did not disappear, but currently I am very loaded with my regular job...)
I've submitted a PR to fix it and another one to test the sctp connectivity, please review them, I've pinged you there :smile:

Related discussion:
text: err := cs.CoreV1().Pods(f.Namespace.Name).Delete(context.TODO(), podName, metav1.DeleteOptions{}) framework.ExpectNoError(err, "failed to delete pod: %s in namespace: %s", podName, f.Namespace.Name) }() <del> ginkgo.By("dumping iptables rules on the node") </del> <ins> // wait until host port manager syncs rules </ins> cmd = "sudo iptables-save" <del> framework.Logf("Executing cmd %q on node %v", cmd, node.Name) result, err := hostExec.IssueCommandWithResult(cmd, node) if err != nil { framework.Failf("Interrogation of iptables rules failed on node %v", node.Name) </del> <ins> if framework.TestContext.ClusterIsIPv6() { cmd = "sudo ip6tables-save" </ins> } <ins> err = wait.PollImmediate(framework.Poll, framework.PollShortTimeout, func() (bool, error) { framework.Logf("Executing cmd %q on node %v", cmd, node.Name) result, err := hostExec.IssueCommandWithResult(cmd, node) if err != nil { framework.Logf("Interrogation of iptables rules failed on node %v", node.Name) return false, nil } </ins> <del> ginkgo.By("checking that iptables contains the necessary iptables rules") found := false for _, line := range strings.Split(result, "n") { if strings.Contains(line, "-p sctp") && strings.Contains(line, "--dport 5060") { found = true break </del> <ins> for _, line := range strings.Split(result, "n") { if strings.Contains(line, "-p sctp") && strings.Contains(line, "--dport 5060") { return true, nil } </ins> } <del> } if !found { </del> <ins> framework.Logf("retrying ... not hostport sctp iptables rules found on node %v", node.Name) return false, nil }) if err != nil { </ins> framework.Failf("iptables rules are not set for a pod with sctp hostport") } ginkgo.By("validating sctp module is still not loaded")