jiebi/CodeConvo huggingface dataset:
Code change:
return utilerrors.NewAggregate(errs) } <del> // Gets the (CPU) container the specified pid is in. </del> <ins> // getContainer returns the cgroup associated with the specified pid. // It enforces a unified hierarchy for memory and cpu cgroups. // On systemd environments, it uses the name=systemd cgroup for the specified pid. </ins> func getContainer(pid int) (string, error) { cgs, err := cgroups.ParseCgroupFile(fmt.Sprintf("/proc/%d/cgroup", pid)) if err != nil { return "", err } <del> cg, ok := cgs["cpu"] if ok { return cg, nil </del> <ins> cpu, found := cgs["cpu"] if !found { return "", cgroups.NewNotFoundError("cpu") } memory, found := cgs["memory"] if !found { return "", cgroups.NewNotFoundError("memory") } // since we use this container for accounting, we need to ensure its a unified hierarchy. if cpu != memory { return "", fmt.Errorf("cpu and memory cgroup hierarchy not unified.  cpu: %s, memory: %s", cpu, memory) } // on systemd, every pid is in a unified cgroup hierarchy (name=systemd as seen in systemd-cgls) // cpu and memory accounting is off by default, users may choose to enable it per unit or globally. // users could enable CPU and memory accounting globally via /etc/systemd/system.conf (DefaultCPUAccounting=true DefaultMemoryAccounting=true). // users could also enable CPU and memory accounting per unit via CPUAccounting=true and MemoryAccounting=true // we only warn if accounting is not enabled for CPU or memory so as to not break local development flows where kubelet is launched in a terminal. // for example, the cgroup for the user session will be something like /user.slice/user-X.slice/session-X.scope, but the cpu and memory // cgroup will be the closest ancestor where accounting is performed (most likely /) on systems that launch docker containers. // as a result, on those systems, you will not get cpu or memory accounting statistics for kubelet. // in addition, you would not get memory or cpu accounting for the runtime unless accounting was enabled on its unit (or globally). if systemd, found := cgs["name=systemd"]; found { if systemd != cpu { glog.Warningf("CPUAccounting not enabled for pid: %d", pid) } if systemd != memory { glog.Warningf("MemoryAccounting not enabled for pid: %d", pid) } return systemd, nil </ins> } <del> return "", cgroups.NewNotFoundError("cpu") </del> <ins> return cpu, nil </ins> } // Ensures the system container is created and all non-kernel threads and process 1

Related discussion:
text: container manager setup node routine should ensure that cpu and memory cgroups for container runtime are of the same value, and warn if they are equal to . For background, on systemd environments, the default unit file for has CPU and Memory accounting not enabled, so the /proc/.../cgroup file reports as cpu cgroup.
/cc - I will send a bug-fix for 1.3 for this in next couple days.
/cc
Looking forward to the fix
The PR that fixes this issue was prematurely merged by the merge bot so re-opening.
Code change:
curNode.ObjectMeta.Labels[kubeletapis.LabelInstanceType] = instanceType } <del> // TODO(wlan0): Move this logic to the route controller using the node taint instead of condition // Since there are node taints, do we still need this? // This condition marks the node as unusable until routes are initialized in the cloud provider if cnc.cloud.ProviderName() == "gce" { curNode.Status.Conditions = append(node.Status.Conditions, v1.NodeCondition{ Type:               v1.NodeNetworkUnavailable, Status:             v1.ConditionTrue, Reason:             "NoRouteCreated", Message:            "Node created without a route", LastTransitionTime: metav1.Now(), }) } </del> if zones, ok := cnc.cloud.Zones(); ok { zone, err := getZoneByProviderIDOrName(zones, curNode) if err != nil {

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: In , the controller add NodeNetworkUnavailable to the node conditions, and use to update it. But node status can not be updated by , it doesn't work because replace the node status with old status. To fix this bug, we should use to update node status. /sig cloud-provider /sig cluster-lifecycle /area cloudprovider What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others:
I agree this is a bug. As a matter of interest though, how are you exercising this bug? It seems specific to cloud controller on gce. However that controller isn't actually enabled on gce. It would be good for me to understand your use case, to make sure we don't break it later.
I built my kubernetes cluster with external cloudprovider, but I found if I stop the route controller, pod will be scheduled on node even if the route is not for that node. I read the code and found that the node condition would not be add when using other cloudproviders. At first, I change "gce" in that line code to the name of my cloudprovider, but the problem also exists. Then I found that didn't take effect actually.
Code change:
// Optional, no events will be sent without it etcdClient tools.EtcdClient <del> // Optional, no statistics will be available if omitted cadvisorClient CadvisorInterface </del> // Optional, defaults to simple implementaiton healthChecker health.HealthChecker // Optional, defaults to simple Docker implementation

Related discussion:
text: So today, cadvisor became wedged and just stopped paying attention to its socket. Not kubelet's fault. But when I would launch all I would get would be: It took me a bit of trolling through netstat and lsof to figure out that the problem was cadvisor being completely locked up. kill -9 cadvisor and everything started working well. I wonder if we should emit more messages during startup about what operation we are about to attempt, not just the results of operations, as I was blaming docker for a while....
kubelet shouldn't block on this. I'll take a look.
Any ideas what was locking cAdvisor? and was it just not responding to requests?
I don't have any idea why it was locked. If I find another one locked I'll hang onto it to debug. Journal had: obviously the 'kill' came from me. It would not respond to systemctl cadvisor stop. netstat looking at port 4194 showed me At which point I realized cadvisor wasn't handling its sockets right since so many long dead kubelet sockets hadn't been cleaned up, so I kill -9'd it.
I cut some interesting info from the cadvisor journal log (trying to cut hostnames) You can see the last message about failed to update stats for container ... happened Oct 08 01:06:01. It was then silent until I killed it Oct 08 19:20:13, even though I'm pretty sure dnf was still being launched every hour on the hour. So it was wedged for about 18 hours.
What kernel is this on? We've seen there is a kernel bug fixed in 3.12+ that sometimes hangs us forever on exit. We're not quite sure how it gets triggered...
16.3
Code change:
} } <ins> // Set the connection reuse mode if val, _ := sysctl.GetSysctl(sysctlConnReuse); val != 0 { if err := sysctl.SetSysctl(sysctlConnReuse, 0); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlConnReuse, err) } } </ins> // Set the ip_forward sysctl we need for if val, _ := sysctl.GetSysctl(sysctlForward); val != 1 { if err := sysctl.SetSysctl(sysctlForward, 1); err != nil {

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Service with clusterIP and a domain , the domain resolve to ingress-nginx. Anothor service will request , when use domain it works fine. I think using clusterIP directly will get better performance because it saves some time for dns resolve and ingress-nginx proxy, so I change domain to clusterIP. But the result was not expected, it got worse, very low throughput and high latency. What you expected to happen: On the above situation, clusterIP should have better performance than ingress-nginx. How to reproduce it (as minimally and precisely as possible): mode, do test with ab Anything else we need to know?: I install an ipvs enviroment manually and do some test, and found it possiblly because ipvs will use iptables to realize ! Environment: Kubernetes version (use ): 1.11.1 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): centos 7.2 Kernel (e.g. ): 3.10 Install tools: binary Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I try to change some kernel param and found is related to this issue , default . change to then complete about 3000k request, pretty good performance. So, should I change the param in production? Does it have any side effects
/area ipvs
Can you look at it?
, default 1. change to 0then complte about 3000k request, pretty good performance. Yes, seems you find the key to the problem - it's set to 0 in my environment and I have not found any issue so far(it's about IPVS module behavior and we will not find parameter in old kernels ). BTW, you can also try to set the ipvs conntrack() to 0 - it will improve performance as well.
Thanks for reply.
Code change:
} } <ins> // Set the connection reuse mode if val, _ := sysctl.GetSysctl(sysctlConnReuse); val != 0 { if err := sysctl.SetSysctl(sysctlConnReuse, 0); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlConnReuse, err) } } </ins> // Set the ip_forward sysctl we need for if val, _ := sysctl.GetSysctl(sysctlForward); val != 1 { if err := sysctl.SetSysctl(sysctlForward, 1); err != nil {

Related discussion:
text: I am using kubernetes , and after set , clusterIP can not be reachable. Which version do I need ?
oops, have you ever configured Cluster-CIDR or iptables masq-all for kube-proxy?
Yes, I configure Cluster-CIDR
Sorry, Cluster IP still needs the IPVS conntrack and nodeport does not rely on it any more - we are trying to figure it out for Cluster IP.
Got it, thanks!
cc Should we tune the kernel parameter in kube-proxy? Given that many users complain about the performance issue, probably the answer is YES.
There is a nice article describing even more kernel parameters to tune ipvs:
Side effect like this comment: <img width="790" alt="hpaprize" src="https://user-"Note: time in picture below is UTC+8. ! Abviously every time scaledown , response time will increase. Any idea for this?
Thanks for reporting. Have you ever read the article posted by In this article, some ipvs parameters can be tuned, e.g.
yes, I had test these parameters not found in centos 7.2(kernel 3.10) will not affect the result is similar with , will be low throughput According to this issue ,I guess it may be the root cause, so I upgrade my setup from 1.11.1 to 1.12.2. But did not solve this problem, and found a new problem, rs delete failed after scaledown. I had open a new issue to this problem:
/assign
Hi, , I raised a pr for this issue. Please check. : ) Looking forward for your advise.
Could you please check out the HEAD to see if your issues have been resolved?
It's not a big surprise this benchmark performs better with . uses http/1.0 so is creating a new connection for every single request. With IPVS will reuse the existing connection entry if source port matches - sending repeated requests to the same real server rather than actually load balancing. So my guess is you will see an unbalanced load with using and , but it makes the benchmark look better. I'm not sure this benchmark is a realistic scenario. I suggest using a tool like instead which uses persistent connections and see what happens.
Apart from the load balancing issue, it's also possible the overhead of is bad when creating connections in a short period of time.
Code change:
} } <ins> // Set the connection reuse mode if val, _ := sysctl.GetSysctl(sysctlConnReuse); val != 0 { if err := sysctl.SetSysctl(sysctlConnReuse, 0); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlConnReuse, err) } } </ins> // Set the ip_forward sysctl we need for if val, _ := sysctl.GetSysctl(sysctlForward); val != 1 { if err := sysctl.SetSysctl(sysctlForward, 1); err != nil {

Related discussion:
text: The benchmark is basically stress testing the new connection handling of IPVS. But I'm not sure we should be optimizing for such a thing, when in reality connections will be much longer lived (like with gRPC, or http/1.1+). It's a trade off between improved load balancing (connreusemode=1 or 2) and connection creation latency (connreusemode=0).
I have a php program use unirest to request other sevice in the cluster. It not use keepalive.
I would prefer to improve the latency number and I think the scenario where one client send lots of requests at the same time is not a common case and mostly happen in test environment.
It might be possible to improve this in the kernel too. You can see the change here: bypasses the reuse checks and expiring logic for new connections.
We set in IPVS proxier in case other people don't know what kernel parameters they should tune. You are familiar with IPVS and linux kernel, but I think many users don't have enough context.
, I think this was because the the container already deleted, but kube-proxy need time to flush the rules, iptables mode should also have this problem.
Yes, I will test preStop recently, may solve the problem
Here the likely reason for the slow throughput is described: As setting to causes , is any of the options listed possible to do? I guess only option 2 might be worth looking at as conntrack is probably needed for the remaining iptables components.
Hello everyone We are very fortunate to tell you that this bug has been fixed by us and has been verified to work very well. The patch is being submitted to the Linux kernel community. You can also apply this patch to your own kernel, and then only need to set (default) and (default). As the sysctl switch is newly . You can adapt the kube-proxy by judging whether there is , if so, it means that the current kernel is the version that fixed this bug.
Code change:
} } <ins> // Set the connection reuse mode if val, _ := sysctl.GetSysctl(sysctlConnReuse); val != 0 { if err := sysctl.SetSysctl(sysctlConnReuse, 0); err != nil { return nil, fmt.Errorf("can't set sysctl %s: %v", sysctlConnReuse, err) } } </ins> // Set the ip_forward sysctl we need for if val, _ := sysctl.GetSysctl(sysctlForward); val != 1 { if err := sysctl.SetSysctl(sysctlForward, 1); err != nil {

Related discussion:
text: That Can solve the following problems: update, IPVS keeps scheduling traffic to the destroyed Pod IPVS traffic scheduling after scaled up or rolling update IPVS low throughput issue second connection delay in masque low throughputÂ Bench can fill up ipvs service proxy in seconds 1s latency in when upgrading from on RHEL 8.1 -proxy ipvs connreuse_mode setting causes errors with high load from single client Thank you. By Yang Yuxi TencentCloudContainerTeam
Following-up on comment above for posterity. The patch mentioned above didn't make it to the kernel but there are two recently merged patches worth highlighting. One of them fixes the 1 second delay issue and the other fixes dropped packets when stale connection entries in the IPVS table are used: 1) 2)
Code change:
kubectl delete secret test-secret --namespace=test-secrets ### Create a tls secret <ins> if [[ "${WAIT_FOR_DELETION:-}" == "true" ]]; then kube::test::wait_object_assert 'secrets --namespace=test-secrets' "{{range.items}}{{$id_field}}:{{end}}" '' fi </ins> # Pre-condition: no SECRET exists kube::test::get_object_assert 'secrets --namespace=test-secrets' "{{range.items}}{{$id_field}}:{{end}}" '' # Command

Related discussion:
text: disabled them since was flaky. We need to find and fix the issue and then reenable these tests. cc Related issues: and
Probably should be higher priority so it doesn't get dropped
The issue is that we dont wait for the resource to be deleted unless we are using a reaper: This breaks for federation since we use finalizers. kubectl should wait for the resource to be deleted. We should also set OrphanDependents = false. This will have no effect in kubernetes but will enable cascading deletion in federation. Ref cc Will send a fix.
Sent
Updated to remove the wait (GET) at the end so it no longer fixes this issue. Sent to update the test to include a wait which fixes this issue.
Code change:
name: cql resources: limits: <del> cpu: "1" </del> <ins> cpu: "500m" </ins> memory: 1Gi requests: <del> cpu: "1" </del> <ins> cpu: "500m" </ins> memory: 1Gi securityContext: capabilities:

Related discussion:
text: https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
sorry wrong issue.
looks like this was recently in #. can you take a look please?
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
This requires persistent storage. Is this flaky on master? Any guidance on how to diagnose this? I did run the e2e tests on GCE.
and is this really a p0?
it's happening pretty frequently, but I don't think we care about examples tests as much.
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
who can assist me with this? I cannot recreate it ... Command I ran. Results that I am getting. Just ran that off of head, and it worked. I am rerunning as the full test suite.
look through the logs, from last test cassandra-2 was stuck in pending which prevents the petset from starting up (c*-3 not even created) you see events like and logs like
so the test is already slow. I am only giving it one cpu, giving it less will make it super slow. Any recommendations? .25 CPU? And where did you find this?
[FLAKE-PING] This flaky-test issue would love to have more attention.
This is going to sit for a bit. With my schedule this week, won't likely get to it.
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Code change:
name: cql resources: limits: <del> cpu: "1" </del> <ins> cpu: "500m" </ins> memory: 1Gi requests: <del> cpu: "1" </del> <ins> cpu: "500m" </ins> memory: 1Gi securityContext: capabilities:

Related discussion:
text: https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
[FLAKE-PING] This flaky-test issue would love to have more attention.
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
https://k8s- Failed: [] [Feature:Example] [] CassandraPetSet should create petset {Kubernetes e2e suite}
[FLAKE-PING] This flaky-test issue would love to have more attention.
Code change:
if accountName == "" || accountKey == "" { return "", "", fmt.Errorf("Invalid %v/%v, couldn't extract azurestorageaccountname or azurestorageaccountkey", nameSpace, secretName) } <ins> accountName = strings.TrimSpace(accountName) </ins> return accountName, accountKey, nil }

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If this may be security issue, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: When trying to mount an Azure File mount it fails, it looks as though its putting a new line between the name of the Azure Storage Account and the Azure storage domain. There are no new line characters in the secret that holds the account name. The following is the secret: The following is the declaration within the section of the Kubernetes yaml: The following is the error that's shown in `kubectl describe po/pod_name: What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: azure OS (e.g. from /etc/os-release): Ubuntu 16.04.3 LTS Kernel (e.g. ): 4.13.0-1011-azure Install tools: Others:
/sig storage
/sig azure
It seems to me that does decode to , so it seems to be behaving correctly. Maybe it's a good idea to trim the return char for and fields after ?
is specified by user, so trim the is not a good idea in the code. Another simple way is to use kubectl create secret to create azure-secret, the accountname and key won't require base64 encoded, so you won't mess up with let me know if you have any question, thanks.
Thanks !
Bumped into this issue today too on our cluster. Can confirm command worked. Is there an easy way to time the /n off the end of the echo for before using base64? Thanks.
yes it's
Of course it is! Doh!
i think would've made better sense to trim the azurestorageaccountname value when receiving it... anyways the storage account name wont have n in them right? I think that just made better developer experience...
is not allowed in azure storage account name
so if azure storage account name does not allow in the name.. then why not just trim the name instead of error... Because devs would just and that would already have the n in the resulting base64-encoded string... right?
sounds reasonable /reopen
Reopened this issue.
Code change:
if accountName == "" || accountKey == "" { return "", "", fmt.Errorf("Invalid %v/%v, couldn't extract azurestorageaccountname or azurestorageaccountkey", nameSpace, secretName) } <ins> accountName = strings.TrimSpace(accountName) </ins> return accountName, accountKey, nil }

Related discussion:
text: <detailsIn response to : sounds reasonable /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
awesome. Thanks
so then this goes in to kube version 1.14.6 in AKS ? or how does this work?
It would only go into k8s v1.16.0 since it's not a critical bug, we don't want to do hot fix to the old release.
got it. thanks Andy.