jiebi/CodeConvo huggingface dataset:
Code change:
// This call is idempotent, and must not return an error if the sandbox has // already been removed. func (f *RemoteRuntime) RemovePodSandbox(ctx context.Context, req *kubeapi.RemovePodSandboxRequest) (*kubeapi.RemovePodSandboxResponse, error) { <del> err := f.RuntimeService.RemovePodSandbox(req.PodSandboxId) </del> <ins> err := f.RuntimeService.StopPodSandbox(req.PodSandboxId) </ins> if err != nil { return nil, err }

Related discussion:
text: </details
/cc
will increase the time for deletion of pod and it should be expected because currently pod is deleted from apiserver before removing the sandbox container but after this fix pod will be deleted from the apiserver only after deleting the sandbox container.
/remove-sig scheduling doesn't affect scheduling running time at all.
Yeah, is almost definitely the culprit. Last time we changed that code path, it also caused a regression. The degradation in performance is expected, but a 5x slowdown is definitely larger than it should be.
do we know if that's because the status used by the path is stale, or if removing sandboxes is actually that slow?
if there's not a quick fix, should we revert and bring it back in once the root issue is understood?
I think a revert is appropriate in this case.
ok, I'll open it
open at
I agree with that 5x is something which shouldn't happen and i can take a look on why thats happening and that might require changing the way how the sandbox is currently deleted which is only during GC.
Code change:
function config-ip-firewall { echo "Configuring IP firewall rules" # The GCI image has host firewall which drop most inbound/forwarded packets. <del> # We need to add rules to accept all TCP/UDP packets. </del> <ins> # We need to add rules to accept all TCP/UDP/ICMP packets. </ins> if iptables -L INPUT | grep "Chain INPUT (policy DROP)" > /dev/null; then <del> echo "Add rules to accept all inbound TCP/UDP packets" </del> <ins> echo "Add rules to accept all inbound TCP/UDP/ICMP packets" </ins> iptables -A INPUT -w -p TCP -j ACCEPT iptables -A INPUT -w -p UDP -j ACCEPT <ins> iptables -A INPUT -w -p ICMP -j ACCEPT </ins> fi if iptables -L FORWARD | grep "Chain FORWARD (policy DROP)" > /dev/null; then <del> echo "Add rules to accept all forwarded TCP/UDP packets" </del> <ins> echo "Add rules to accept all forwarded TCP/UDP/ICMP packets" </ins> iptables -A FORWARD -w -p TCP -j ACCEPT iptables -A FORWARD -w -p UDP -j ACCEPT <ins> iptables -A FORWARD -w -p ICMP -j ACCEPT </ins> fi }

Related discussion:
text: ref:
identified one problem: firewall is not open for icmp in GCI setup. I opened it manually. Still does not work.
Here are the symptoms: node can ping each other one node, it can ping all pods on it, and vice versa. one node or pod, it can ping all cbr0 gateways(10.X.X.1). pod, it cannot ping but able to curl. It cannot ping any other pod, even on the same bridge, but able to curl.
My first thought would be somethings blockign ping. your options are iptables, firewall, icmp is disabled alltogether (sysctl or somewhere in /proc/sys/net/ there should be a setting). Try traceroute and see where it gets to.
I have a working version now. GCI setup icmp ACCEPT rules in INPUT chain of filter table. But not in OUTPUT and FORWARD chain.
In summary, 2 problems: is not opened for icmp is filtering icmp on FORWARD chain. ACCEPT on INPUT chain is not enough Will submit a fix
Great detective work, Is this a ping-specific problem, or are there going to be problems with other types of outgoing traffic from pods on GCI as well?
I do not see any problem for udp/tcp traffic so far. e2e tests already covers lots of them. Unfortunately, no e2e covers ping tests and GCI happens to filter it.
We should add a test in networking suite for ping.
Is this missing a milestone 1.3?
Code change:
return result, nil } <del> // logPluginScores adds summarized plugin score logging. The goal of this block is to show the highest scoring plugins on // each node, and the average score for those plugins across all nodes. func logPluginScores(nodes []*v1.Node, scoresMap framework.PluginToNodeScores, pod *v1.Pod) { totalPluginScores := make(map[string]int64) for j := range scoresMap { for i := range nodes { totalPluginScores[j] += scoresMap[j][i].Score } } // Build a map of Nodes->PluginScores on that node nodeToPluginScores := make(framework.PluginToNodeScores, len(nodes)) for _, node := range nodes { nodeToPluginScores[node.Name] = make(framework.NodeScoreList, len(scoresMap)) } // Convert the scoresMap (which contains Plugins->NodeScores) to the Nodes->PluginScores map for plugin, nodeScoreList := range scoresMap { for _, nodeScore := range nodeScoreList { klog.V(10).InfoS("Plugin scored node for pod", "pod", klog.KObj(pod), "plugin", plugin, "node", nodeScore.Name, "score", nodeScore.Score) nodeToPluginScores[nodeScore.Name] = append(nodeToPluginScores[nodeScore.Name], framework.NodeScore{Name: plugin, Score: nodeScore.Score}) } } for node, scores := range nodeToPluginScores { // Get the top 3 scoring plugins for each node sort.Slice(scores, func(i, j int) bool { return scores[i].Score > scores[j].Score }) var topScores []pluginScores for _, score := range scores { pluginScore := pluginScores{ Plugin:           score.Name, Score:            score.Score, AverageNodeScore: float64(totalPluginScores[score.Name]) / float64(len(nodes)), } topScores = append(topScores, pluginScore) if len(topScores) == 3 { break } } klog.InfoS("Top 3 plugins for pod on node", "pod", klog.KObj(pod), "node", node, "scores", topScores, ) } } </del> // NewGenericScheduler creates a genericScheduler object. func NewGenericScheduler( cache internalcache.Cache,

Related discussion:
text: Scheduler throughput drops significantly in http://perf- ! After , seems to be the curlspirit.
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent /sig scheduling /sig scalability
Code change:
// Returns true if and only if changes were made // If the security group no longer exists, will return (false, nil) func (c *Cloud) removeSecurityGroupIngress(securityGroupID string, removePermissions []*ec2.IpPermission) (bool, error) { <ins> // We do not want to make changes to the Global defined SG if securityGroupID == c.cfg.Global.ElbSecurityGroup { return false, nil } </ins> group, err := c.findSecurityGroup(securityGroupID) if err != nil { glog.Warningf("Error retrieving security group: %q", err)

Related discussion:
text: /kind bug What happened: When setting an elbSecurityGroup in for AWS, and deploying multiple services with differing ports, then the security group rules flap in AWS, causing services to become inaccessible (only one will be correctly configured at a time). What you expected to happen: I expect that the configured elbSecurityGroup's rules will not be touched at all. It must be pre-configured to permit access. Alternatively, all ports for all services are queried and managed together. How to reproduce it (as minimally and precisely as possible): See above. Anything else we need to know?: Most people are not running large clusters which may hit the AWS limits, so they won't use this option, nor see this issue. Environment: Kubernetes version (use ): 1.7.1 Cloud provider or hardware configuration**: AWS OS (e.g. from /etc/os-release): CentOS 7 Kernel (e.g. ): 4.10.13-1-ARCH Install tools: Custom Others:
There are no sig labels on this issue. Please by: a sig: e.g., to notify the contributor experience sig, OR the label manually: e.g., to apply the label Note: Method 1 will trigger an email to the group. You can find the group list and label list . The in the method 1 has to be replaced with one of these: bugs, feature-requests, pr-reviews, test-failures, proposals
/sig aws
This is fixed by
Code change:
// pod stats api summarizes ephemeral storage usage (container, emptyDir, host[etc-hosts, logs]) podEphemeralStorageTotalUsage := &resource.Quantity{} <del> if podStats.EphemeralStorage != nil { </del> <ins> if podStats.EphemeralStorage != nil && podStats.EphemeralStorage.UsedBytes != nil { </ins> podEphemeralStorageTotalUsage = resource.NewQuantity(int64(*podStats.EphemeralStorage.UsedBytes), resource.BinarySI) } podEphemeralStorageLimit := podLimits[v1.ResourceEphemeralStorage]

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: ! What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): v1.20.1 Cloud provider or hardware configuration: OS (e.g: ): 7 Kernel (e.g. ): Linux zsm-test-2 3.10.0-1160.11.1.el7.x8664 SMP Fri Dec 18 16:34:56 UTC 2020 x8664 x8664 x8664 GNU/Linux Install tools: Network plugin and version (if this is a network-related bug): Others:
/sig node
Do you know how to reproduce this problem?
Yes, can reproduce the problem stably, but the steps are a little complicated.
/triage accepted
Code change:
if err != nil { klog.Warningf("error looking for nvme volume %q: %v", volumeID, err) } else if nvmePath != "" { <ins> if partition != "" { nvmePath = nvmePath + nvmeDiskPartitionSuffix + partition } </ins> devicePaths = append(devicePaths, nvmePath) } }

Related discussion:
text: When using a AWS EBS PV or on-line volume with a partition on a Nitro machine, the volume is available as NVMe device. Kubelet is not able to format & mount the partition: It seems to me that AWS EBS volume plugin does not support partitions at all. The partition is mounted. Using an in-line volume with this pod:
/sig storage /sig cloud-provider /area provider/aws
/triage accepted
Code change:
// back to the caller. in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPod := &Pod{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, </ins> } <ins> SetObjectDefaults_Pod(tmpPod) in.Spec.InitContainers = tmpPod.Spec.InitContainers </ins> } // If there is a beta annotation, copy to alpha key. // See commit log for PR #31026 for why we do this.

Related discussion:
text: What happened: kubelet doesn't seem to be able to manage containers started with CRI-O any more I'm testing kubelet + CRI-O on an Ubuntu 18.04 server installation, inside a qemu/kvm VM managed by Vagrant (see ), in order to be able to test k8s + KubeVirt + KataContainers, which are all compatible with CRI-O kubelet will obsess on starting new containers, which looks like and endless process, and the cluster never gets ready. For instance here are some syslogs : What you expected to happen: It used to work (see asciinema recording at ) a few days ago, but I fear some recent changes broke it. How to reproduce it (as minimally and precisely as possible): Testing with vagrant up --provider=libvirt on Linux with a clone of that commit: vagrant ssh into the VM and ps -edf f or tail -f /var/log/syslog would do... Anything else we need to know?: Environment: Kubernetes version (use ):Cloud provider or hardware configuration: qemu/kvm VM +OS (e.g: ): Ubuntu 18.04.2 LTS Kernel (e.g. ): Linux kubernetes-vagrant-01 4.15.0-48-generic -Ubuntu SMP Wed Apr 3 08:28:49 UTC 2019 x8664 x8664 x8664 GNU/Linux Install tools: kubeadm as in Network plugin and version (if this is a network-related bug): Others:
/sig node
I can confirm this. If I start cri-o from a systemd service it fails with the above mentioned error message. Starting crio with without systemd works just fine. systemd file:
Thanks for your detailed report :) Just want to do some quick version confirming - you've been using k8s 1.14.1 both when it did and did not work? Any other version changes?
Not sure for before, when it used to work, but according to the asciinema recording mentioned above (), it seems the same. However, not sure for CRI-O, underlying OS stuff (systemd, kernel, etc.). Sorry.
I was looking for some information for days. I have exactly the same issue when I use podman + systemd on 18.04.2.
Code change:
// back to the caller. in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPod := &Pod{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, </ins> } <ins> SetObjectDefaults_Pod(tmpPod) in.Spec.InitContainers = tmpPod.Spec.InitContainers </ins> } // If there is a beta annotation, copy to alpha key. // See commit log for PR #31026 for why we do this.

Related discussion:
text: It is the same for both 14.15 and 14.18 kernels. Podman without systemd runs fine. Podman with systemd fails at runc with the same error podman[]: open /dev/stderr: no such device or address podman[]: NAME: runc - Open Container Initiative runtime podman[]: runc is a command line client for running applications packaged according to ... podman[]: time="2019-05-11T13:04:59-07:00" level=error msg="open /dev/stderr: no such device or addressn" podman[]: open /dev/stderr: no such device or address
I've tried to reproduce it on 18.04.2 using Podman from but I am not able to reproduce it. Do you have any reproducer? How did you install Podman?
I used ceph unit files to run ceph in podman rather than docker. Systemd unit successfully starts docker but fails with podman. I can successfully run the same podman container run command from the command line. How were you trying to replicate this? Can you post your unit file?
Hi, I'am having the same problem with Kubernetes with CRI-O. After joining a second worker node it fails to start flannel. gives the following error: What I've tried: Installing previous version of Kubernetes (from to ) Installing previous version of crio (from to ) Building runc myself from their github page and selecting this in the Running it on Ubunut 18.04.2 Issue: (the one giuseppe linked)
I wonder if the issue is with systemd and runc. In both cases, crio and podman, I see runc help output which tells me that something is not being passed to runc that it needs.
I've only used so it is different than your test and you confirmed it works from the CLI for you as well. Could you share your service file so I test it?
I've tried systemd and cgroupfs but no luck so far.
so it seems that runc fails to re-open . That seems like a runc issue, as it doesn't really need to reopen it. To fix the failure, you could use something like:
Just an example.
Code change:
// back to the caller. in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPod := &Pod{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, </ins> } <ins> SetObjectDefaults_Pod(tmpPod) in.Spec.InitContainers = tmpPod.Spec.InitContainers </ins> } // If there is a beta annotation, copy to alpha key. // See commit log for PR #31026 for why we do this.

Related discussion:
text: I use ceph-ansible to install cluster so it may need to install additional daemons like mon before running mgr but the error, in this case, should be different, [Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  {{ cephmgrdockerextraenv }}  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- delighted <:
BTW, I projectatomic ppa and used apt-get -y podman to install. My podman version is 1.3.0 Sasha Litvak — You are receiving this because you commented. Reply to this email directly, view it on GitHub , or mute the thread
Corrected file.
Code change:
// back to the caller. in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPod := &Pod{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, </ins> } <ins> SetObjectDefaults_Pod(tmpPod) in.Spec.InitContainers = tmpPod.Spec.InitContainers </ins> } // If there is a beta annotation, copy to alpha key. // See commit log for PR #31026 for why we do this.

Related discussion:
text: [Unit] Description=Ceph Manager [Service] EnvironmentFile=-/etc/environment ExecStartPre=-/usr/bin/podman stop ceph-mgr-storage2n1-la ExecStartPre=-/usr/bin/podman rm ceph-mgr-storage2n1-la ExecStart=/usr/bin/podman run --rm --net=host  --memory=16G  --cpus=4  -v /var/lib/ceph:/var/lib/ceph:z  -v /etc/ceph:/etc/ceph:z  -v /var/run/ceph:/var/run/ceph:z  -v /etc/localtime:/etc/localtime:ro  -e CLUSTER=ceph  -e CEPHDAEMON=MGR  -e CONTAINERIMAGE=  --name=ceph-mgr-storage2n1-la  ExecStopPost=-/usr/bin/podman stop ceph-mgr-storage2n1-la Restart=always RestartSec=10s TimeoutStartSec=120 TimeoutStopSec=15 [Install] WantedBy=multi- Giuseppe Scrivano <:
Would it have an effect system wide? Can I add it to a unit file or unit override instead? Giuseppe Scrivano <:
yes, I think it is better to add it only to the unit file
Why would runc work outside of systemd? And if it needs stderr, giving it Null in settings will redirect stderr to Null I presume? On Tue, May 14, 2019, 9:46 AM Giuseppe Scrivano <:
Running into same issues on Ubuntu 18.04.2 using cri-o-1.13 via PPA.
PR here: opencontainers/runc
Until it is avaialble with project atomic and such, can I use a null workaround? Giuseppe Scrivano <:
yes, but you will lose anything written to stderr. You could probably also use a file, like
What about actually mounting dev and using /dev/stderr ? On Tue, May 14, 2019, 3:11 PM Giuseppe Scrivano <:
is a symlink to . It is present, the issue here is that is a socket. The fix for runc got merged, could you use an updated runc?
I tried apt update/upgrade and restarting crio, but I'm still getting the issue.
Code change:
// back to the caller. in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPod := &Pod{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, </ins> } <ins> SetObjectDefaults_Pod(tmpPod) in.Spec.InitContainers = tmpPod.Spec.InitContainers </ins> } // If there is a beta annotation, copy to alpha key. // See commit log for PR #31026 for why we do this.

Related discussion:
text: Could it be that crio's runc is not updated yet? Edit Build new runc from and it is working now. Think crio hasn't updated it yet with their repository.
I'm running this version from PPA repo for ubuntu and it fails: could you point to version/commit or whatever helps identify the fixed version/snapshot ? Thanks in advance.
at the moment it is only fixed upstream. You'd need to build runc manually
Is there a version of the package on ppa we can downgrade to? On Wed, May 15, 2019, 4:35 AM Giuseppe Scrivano <:
could we update the runc version in the PPA repo?
Thank you. I seem them built but nut pushed to ppa yet.
should be available now.. let me know..
cri-o-runc (1.0.0-rc6-1~dev~ubuntu18.04~ppa42) bionic; urgency=medium autobuilt -- Lokesh Mandvekar (Bot) <, 15 May 2019 16:54:40 +0000
Thank you, everyone, for a quick turn around. Quick testing showed that fix is working, but more testing is needed. Lokesh Mandvekar <:
I confirm my initial tests with seem to confirm the regression is fixed. Many thanks
I also confirm the fix working for podman. Olivier Berger <:
Code change:
// back to the caller. in.Spec.InitContainers = values // Call defaulters explicitly until annotations are removed <del> for i := range in.Spec.InitContainers { c := &in.Spec.InitContainers[i] SetDefaults_Container(c) </del> <ins> tmpPod := &Pod{ Spec: PodSpec{ HostNetwork:    in.Spec.HostNetwork, InitContainers: values, }, </ins> } <ins> SetObjectDefaults_Pod(tmpPod) in.Spec.InitContainers = tmpPod.Spec.InitContainers </ins> } // If there is a beta annotation, copy to alpha key. // See commit log for PR #31026 for why we do this.

Related discussion:
text: <!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): No What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): initContainers imagePullPolicy initContainers required value Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG REPORT <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: GCE OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: cluster/kube- Others: What happened: When trying to create either a ReplicaSet or Replication Controller whose Pod template contains an Init Container, the validation fails saying is a required value. What you expected to happen: The create should succeed and the Init Container should receive a default . How to reproduce it (as minimally and precisely as possible): Try to create either of the following: ReplicaSet Replication Controller Anything else do we need to know: If I add to the Init Container in the ReplicaSet or Replication Controller Pod template, the create succeeds. When I create an Init Container without in a StatefulSet Pod template, it succeeds. When I create an Init Container without in a bare Pod, it succeeds:
Looks like defaulting not being applied on server? That is probably due to removing defaulting from convert. Surprised we didn't catch this earlier
Some values still not being defaulted
, I'm going to create a PR if none works on it.
We decode from bytes, set defaults and then convert; but for init containers, we delay the decode from "decode phase" to "convert phase" at .