jiebi/CodeConvo huggingface dataset:
Code change:
// set status for Pods created on versions of kube older than 1.6 apiPodStatus.QOSClass = v1qos.GetPodQOS(pod) <ins> oldPodStatus, found := kl.statusManager.GetPodStatus(pod.UID) if !found { oldPodStatus = pod.Status } </ins> apiPodStatus.ContainerStatuses = kl.convertToAPIContainerStatuses( pod, podStatus, <del> pod.Status.ContainerStatuses, </del> <ins> oldPodStatus.ContainerStatuses, </ins> pod.Spec.Containers, len(pod.Spec.InitContainers) > 0, false, ) apiPodStatus.InitContainerStatuses = kl.convertToAPIContainerStatuses( pod, podStatus, <del> pod.Status.InitContainerStatuses, </del> <ins> oldPodStatus.InitContainerStatuses, </ins> pod.Spec.InitContainers, len(pod.Spec.InitContainers) > 0, true,

Related discussion:
text: After , we observed pods moving from phase to . Presents on master and release-1.8 as it was picked in test script has an command for basically doing a CLI watch or you can look in the kubelet log to see the phase transition from to Thanks to for helping isolate this :+1: /sig node
/assign
taking a look. although at first glance it is strange this didnt happen with evicted pods prior to this.
Thanks. FYI, I did confirm that reverting the mentioned PR on master corrected the issue.
This showed up breaking a job controller that depended on the pod not transitioning back to pending. It's possible this can cause applications to fail / run multiple times in potentially dangerous ways. I'm not sure we should allow a status update that moves back to a non terminal state via validation. It's not a legal transition as we have defined phase. We should separately put a guard in place that will report a clear error to clients.
after thinking on it, I think that the situation with eviction is a little different in that the containers are terminated but they are not immediately cleaned up. It is the proximity between pod termination time and container removal time that is uncovering the issue, I think. AFAICT, some path is attempting to get the container status after the container has been removed. When this fails, something in the status manager gets messed up and it under the impression that the pod is new and the status needs to be reinitialized.
Here are the relevant logs showing the bug: Still trying to figure out what caused the status update.
Ok, I figured out why this is happening. When we compute a container's status, we assume that . So deleting a pod's containers before the pod object is removed causes the state to return to pending.
Maybe we should expand this that I to prevent init container status reset to include all containers with an oldStatus
Would that prevent updating a container that crashed and restarted? Edit: nvm, I think that might work
I a comment at
i also commented here that it would be good if syncPod in status manager protected against an invalid transfer of phase... see: can we do that in this same PR?
My hope is to cherrypick this to 1.8. Is that check something we want to cherrypick as well?
Code change:
// set status for Pods created on versions of kube older than 1.6 apiPodStatus.QOSClass = v1qos.GetPodQOS(pod) <ins> oldPodStatus, found := kl.statusManager.GetPodStatus(pod.UID) if !found { oldPodStatus = pod.Status } </ins> apiPodStatus.ContainerStatuses = kl.convertToAPIContainerStatuses( pod, podStatus, <del> pod.Status.ContainerStatuses, </del> <ins> oldPodStatus.ContainerStatuses, </ins> pod.Spec.Containers, len(pod.Spec.InitContainers) > 0, false, ) apiPodStatus.InitContainerStatuses = kl.convertToAPIContainerStatuses( pod, podStatus, <del> pod.Status.InitContainerStatuses, </del> <ins> oldPodStatus.InitContainerStatuses, </ins> pod.Spec.InitContainers, len(pod.Spec.InitContainers) > 0, true,

Related discussion:
text: i dont see why not... absent that its hard to know this fix actually fixes it... or that we dont regress.
do we want to do it in or ? is on the other side of a channel where a stack trace on the caller of could be more informative.
both? i just know that we get the latest state as told to api server on the other end of the channel...
Code change:
return t.delegate.New(kind) } } <ins> func NewThirdPartyParameterCodec(p runtime.ParameterCodec) runtime.ParameterCodec { return &thirdPartyParameterCodec{p} } type thirdPartyParameterCodec struct { delegate runtime.ParameterCodec } func (t *thirdPartyParameterCodec) DecodeParameters(parameters url.Values, from unversioned.GroupVersion, into runtime.Object) error { return t.delegate.DecodeParameters(parameters, v1.SchemeGroupVersion, into) } func (t *thirdPartyParameterCodec) EncodeParameters(obj runtime.Object, to unversioned.GroupVersion) (url.Values, error) { return t.delegate.EncodeParameters(obj, v1.SchemeGroupVersion) } </ins>

Related discussion:
text: A user report that it's broken in the slack channel:
Also and -- this probably broke with one of our recent refactors. The server should be accepting list options from any version, I thought?
FWIW, I think it's this line barf at the ListOptions: The parameter codec first decodes the query parameter to a versioned object and then convert it to the internal object. The code fails because it cannot create a "" ListOptions.
The server should accept list options in the API version it is sent. ListOptions should be registered in all third party groups.
The thirdPartyResourceDataCreator should handle "ListOptions", which surprises me that it doesn't then respect the group.
I think we should add a wrapper around the api.ParameterCodec here: Do something similar as what does.
Putting together a PR now and will let you guys review.
I guess my point is, the list options are just there from the URL query parameters. The user didn't send a group. Why do we assume that value for the group rather than "/v1" or something?
My understanding is we want to allow options to evolve differently in different groups. We do have v1.ListOptions and v1beta1.ListOptions. And if a client sends requeset to extensions/v1beta1 endpoint, then it's expected to encode the ListOptions to v1beta1 and then convert it to query parameters.
Mostly because the parameter codec is supposed to abstract that. "virtual" groups need to be implemented by the virtual "parameter codec". The statement on floating server objects (List, Delete etc) was that they have a locked API schema, and we have a mechanism to do that via the scheme which is registering them in a group. , Daniel Smith wrote:
Code change:
* A taint consists of a key, value, and effect. As an argument here, it is expressed as key=value:effect. * The key must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores, up to %[1]d characters. * The value must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores, up to %[2]d characters. <del> * The effect must be NoSchedule or PreferNoSchedule. </del> <ins> * The effect must be NoSchedule, PreferNoSchedule or NoExecute. </ins> * Currently taint can only apply to node.`) taint_example = templates.Examples(`

Related discussion:
text: <!-- Thanks for filing an issue! Before hitting the button, please answer these questions.--Is this a request for help? (If yes, you should use our troubleshooting guide and community support channels, see ): What keywords did you search in Kubernetes issues before filing this one? (If you have found any duplicates, you should instead reply there.): Is this a BUG REPORT or FEATURE REQUEST? (choose one): <!-- If this is a BUG REPORT, please: Fill in as much of the template below as you can. If you leave out information, we can't help you as well. If this is a FEATURE REQUEST, please: Describe in detail the feature/behavior/change you'd like to see. In both cases, be ready for followup questions, and please respond in a timely manner. If we can't reproduce a bug or think a feature already exists, we might close your issue. If we're wrong, PLEASE feel free to reopen it and explain why. --Kubernetes version (use ): Environment: Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): Kernel (e.g. ): Install tools: Others: What happened: The help for needs to be updated to reflect the latest change. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know:
cc I will work on this together with
Code change:
const sysctlRouteLocalnet = "net/ipv4/conf/all/route_localnet" const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables" const sysctlVSConnTrack = "net/ipv4/vs/conntrack" <ins> const sysctlConnReuse = "net/ipv4/vs/conn_reuse_mode" </ins> const sysctlForward = "net/ipv4/ip_forward" const sysctlArpIgnore = "net/ipv4/conf/all/arp_ignore" const sysctlArpAnnounce = "net/ipv4/conf/all/arp_announce"

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks!--What happened: Service with clusterIP and a domain , the domain resolve to ingress-nginx. Anothor service will request , when use domain it works fine. I think using clusterIP directly will get better performance because it saves some time for dns resolve and ingress-nginx proxy, so I change domain to clusterIP. But the result was not expected, it got worse, very low throughput and high latency. What you expected to happen: On the above situation, clusterIP should have better performance than ingress-nginx. How to reproduce it (as minimally and precisely as possible): mode, do test with ab Anything else we need to know?: I install an ipvs enviroment manually and do some test, and found it possiblly because ipvs will use iptables to realize ! Environment: Kubernetes version (use ): 1.11.1 Cloud provider or hardware configuration: OS (e.g. from /etc/os-release): centos 7.2 Kernel (e.g. ): 3.10 Install tools: binary Others: <!-- DO NOT EDIT BELOW THIS LINE --/kind bug
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I try to change some kernel param and found is related to this issue , default . change to then complete about 3000k request, pretty good performance. So, should I change the param in production? Does it have any side effects
/area ipvs
Can you look at it?
, default 1. change to 0then complte about 3000k request, pretty good performance. Yes, seems you find the key to the problem - it's set to 0 in my environment and I have not found any issue so far(it's about IPVS module behavior and we will not find parameter in old kernels ). BTW, you can also try to set the ipvs conntrack() to 0 - it will improve performance as well.
Thanks for reply.
Code change:
const sysctlRouteLocalnet = "net/ipv4/conf/all/route_localnet" const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables" const sysctlVSConnTrack = "net/ipv4/vs/conntrack" <ins> const sysctlConnReuse = "net/ipv4/vs/conn_reuse_mode" </ins> const sysctlForward = "net/ipv4/ip_forward" const sysctlArpIgnore = "net/ipv4/conf/all/arp_ignore" const sysctlArpAnnounce = "net/ipv4/conf/all/arp_announce"

Related discussion:
text: I am using kubernetes , and after set , clusterIP can not be reachable. Which version do I need ?
oops, have you ever configured Cluster-CIDR or iptables masq-all for kube-proxy?
Yes, I configure Cluster-CIDR
Sorry, Cluster IP still needs the IPVS conntrack and nodeport does not rely on it any more - we are trying to figure it out for Cluster IP.
Got it, thanks!
cc Should we tune the kernel parameter in kube-proxy? Given that many users complain about the performance issue, probably the answer is YES.
There is a nice article describing even more kernel parameters to tune ipvs:
Side effect like this comment: <img width="790" alt="hpaprize" src="https://user-"Note: time in picture below is UTC+8. ! Abviously every time scaledown , response time will increase. Any idea for this?
Thanks for reporting. Have you ever read the article posted by In this article, some ipvs parameters can be tuned, e.g.
yes, I had test these parameters not found in centos 7.2(kernel 3.10) will not affect the result is similar with , will be low throughput According to this issue ,I guess it may be the root cause, so I upgrade my setup from 1.11.1 to 1.12.2. But did not solve this problem, and found a new problem, rs delete failed after scaledown. I had open a new issue to this problem:
/assign
Hi, , I raised a pr for this issue. Please check. : ) Looking forward for your advise.
Could you please check out the HEAD to see if your issues have been resolved?
It's not a big surprise this benchmark performs better with . uses http/1.0 so is creating a new connection for every single request. With IPVS will reuse the existing connection entry if source port matches - sending repeated requests to the same real server rather than actually load balancing. So my guess is you will see an unbalanced load with using and , but it makes the benchmark look better. I'm not sure this benchmark is a realistic scenario. I suggest using a tool like instead which uses persistent connections and see what happens.
Apart from the load balancing issue, it's also possible the overhead of is bad when creating connections in a short period of time.
Code change:
const sysctlRouteLocalnet = "net/ipv4/conf/all/route_localnet" const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables" const sysctlVSConnTrack = "net/ipv4/vs/conntrack" <ins> const sysctlConnReuse = "net/ipv4/vs/conn_reuse_mode" </ins> const sysctlForward = "net/ipv4/ip_forward" const sysctlArpIgnore = "net/ipv4/conf/all/arp_ignore" const sysctlArpAnnounce = "net/ipv4/conf/all/arp_announce"

Related discussion:
text: The benchmark is basically stress testing the new connection handling of IPVS. But I'm not sure we should be optimizing for such a thing, when in reality connections will be much longer lived (like with gRPC, or http/1.1+). It's a trade off between improved load balancing (connreusemode=1 or 2) and connection creation latency (connreusemode=0).
I have a php program use unirest to request other sevice in the cluster. It not use keepalive.
I would prefer to improve the latency number and I think the scenario where one client send lots of requests at the same time is not a common case and mostly happen in test environment.
It might be possible to improve this in the kernel too. You can see the change here: bypasses the reuse checks and expiring logic for new connections.
We set in IPVS proxier in case other people don't know what kernel parameters they should tune. You are familiar with IPVS and linux kernel, but I think many users don't have enough context.
, I think this was because the the container already deleted, but kube-proxy need time to flush the rules, iptables mode should also have this problem.
Yes, I will test preStop recently, may solve the problem
Here the likely reason for the slow throughput is described: As setting to causes , is any of the options listed possible to do? I guess only option 2 might be worth looking at as conntrack is probably needed for the remaining iptables components.
Hello everyone We are very fortunate to tell you that this bug has been fixed by us and has been verified to work very well. The patch is being submitted to the Linux kernel community. You can also apply this patch to your own kernel, and then only need to set (default) and (default). As the sysctl switch is newly . You can adapt the kube-proxy by judging whether there is , if so, it means that the current kernel is the version that fixed this bug.
Code change:
const sysctlRouteLocalnet = "net/ipv4/conf/all/route_localnet" const sysctlBridgeCallIPTables = "net/bridge/bridge-nf-call-iptables" const sysctlVSConnTrack = "net/ipv4/vs/conntrack" <ins> const sysctlConnReuse = "net/ipv4/vs/conn_reuse_mode" </ins> const sysctlForward = "net/ipv4/ip_forward" const sysctlArpIgnore = "net/ipv4/conf/all/arp_ignore" const sysctlArpAnnounce = "net/ipv4/conf/all/arp_announce"

Related discussion:
text: That Can solve the following problems: update, IPVS keeps scheduling traffic to the destroyed Pod IPVS traffic scheduling after scaled up or rolling update IPVS low throughput issue second connection delay in masque low throughputÂ Bench can fill up ipvs service proxy in seconds 1s latency in when upgrading from on RHEL 8.1 -proxy ipvs connreuse_mode setting causes errors with high load from single client Thank you. By Yang Yuxi TencentCloudContainerTeam
Following-up on comment above for posterity. The patch mentioned above didn't make it to the kernel but there are two recently merged patches worth highlighting. One of them fixes the 1 second delay issue and the other fixes dropped packets when stale connection entries in the IPVS table are used: 1) 2)
Code change:
func BindContextFlags(contextInfo *clientcmdapi.Context, flags *pflag.FlagSet, flagNames ContextOverrideFlags) { flagNames.ClusterName.BindStringFlag(flags, &contextInfo.Cluster) flagNames.AuthInfoName.BindStringFlag(flags, &contextInfo.AuthInfo) <del> flagNames.Namespace.BindStringFlag(flags, &contextInfo.Namespace) </del> <ins> flagNames.Namespace.BindTransformingStringFlag(flags, &contextInfo.Namespace, RemoveNamespacesPrefix) } // RemoveNamespacesPrefix is a transformer that strips "ns/", "namespace/" and "namespaces/" prefixes case-insensitively func RemoveNamespacesPrefix(value string) (string, error) { for _, prefix := range []string{"namespaces/", "namespace/", "ns/"} { if len(value) > len(prefix) && strings.EqualFold(value[0:len(prefix)], prefix) { value = value[len(prefix):] break } } return value, nil </ins> }

Related discussion:
text: Because always return names with a prefix like , it cannot be directly used to iterate over all namespaces to execute other commands. I've compiled from source: Reviewing some scripts, I came across a usage of like this: That line fetches namespace names to later run in a loop: That seemed to be a perfect use case for the flag, but it can't be used without extra hacks to get rid of the prefix (FWIW, is invalid). Changing the current output of might not be possible or desirable, for this could impact existing users that rely on the output format. If I'm wrong about the above limitation, than it would make sense to output just the name when and there is a single resource type, while using the ideas introduced in to show the resource type only when there is more than one type in the output, or when is explicitly set. Before: After:
could you help me ping the appropriate team to triage this? Thanks!
Please re-post your question to . We are trying to consolidate the channels to which questions for help/support are posted so that we can improve our efficiency in responding to your requests, and to make it easier for you to find answers to frequently asked questions and how to address common use cases. We regularly see messages posted in multiple forums, with the full response thread only in one place or, worse, spread across multiple forums. Also, the large volume of support issues on github is making it difficult for us to use issues to identify real bugs. The Kubernetes team scans stackoverflow on a regular basis, and will try to ensure your questions don't go unanswered. Before posting a new question, please search stackoverflow for answers to similar questions, and also familiarize yourself with: Again, thanks for using Kubernetes. The Kubernetes Team
perhaps I didn't do a good job at explaining the problem, but this is not a question of how to do something, but a report on a problem with the CLI design.
Sorry about that :-/. I'll try to route to the proper team.
update: console examples to the description.
the -o format is designed for scripting, the use case is just different from yours. You should be able to paste the output of into . Does do what you want? If this is a common use case we could create another -o flag value.
Code change:
func BindContextFlags(contextInfo *clientcmdapi.Context, flags *pflag.FlagSet, flagNames ContextOverrideFlags) { flagNames.ClusterName.BindStringFlag(flags, &contextInfo.Cluster) flagNames.AuthInfoName.BindStringFlag(flags, &contextInfo.AuthInfo) <del> flagNames.Namespace.BindStringFlag(flags, &contextInfo.Namespace) </del> <ins> flagNames.Namespace.BindTransformingStringFlag(flags, &contextInfo.Namespace, RemoveNamespacesPrefix) } // RemoveNamespacesPrefix is a transformer that strips "ns/", "namespace/" and "namespaces/" prefixes case-insensitively func RemoveNamespacesPrefix(value string) (string, error) { for _, prefix := range []string{"namespaces/", "namespace/", "ns/"} { if len(value) > len(prefix) && strings.EqualFold(value[0:len(prefix)], prefix) { value = value[len(prefix):] break } } return value, nil </ins> }

Related discussion:
text: the use cases I've observed doing code reviews: over all namespaces to get certain resources from that namespace: over all namespaces, then iterate of over resource types that produce logs and fetch logs one by one: Above we can use for listing pods because later we'll pass it to that accept the syntax. We cannot do the same for namespaces, because doesn't like the format.
Alternatively we could be kubectl recognize as valid
I'll take a look on it if it's not taken by anyone yet. CC
Have you started this?
Only investigated due to being out of office since Thursday last week. I can work on it in upcoming week, but I'll understand if it's piority issue and you'll reassign it.
Ready :-)
There are no sig labels on this issue. Please by:<br(1) mentioning a sig: <br(2) specifying the label manually: <br<brNote: method (1) will trigger a notification to the team. You can find the team list .
Reiterating the mentions to trigger a notification: <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
func getRandomClusterZone(c clientset.Interface) string { zones, err := e2enode.GetClusterZones(c) <ins> zone := "" </ins> framework.ExpectNoError(err) <del> framework.ExpectNotEqual(len(zones), 0) zonesList := zones.UnsortedList() return zonesList[rand.Intn(zones.Len())] </del> <ins> if len(zones) != 0 { zonesList := zones.UnsortedList() zone = zonesList[rand.Intn(zones.Len())] } return zone </ins> }

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --If the provide is , run the below storage testcase, Hit below exception, At first glance, it looks like the return value is not correct, but looking into how the testcases was wrote, the root case should be that the testcase below doesn't support the provide, SSD PD on GCE/GKE gp2 EBS on AWS We should provide the correct error message which will be useful for trouble shooting instead. Kubernetes version (use ): All Cloud provider or hardware configuration: local OS (e.g: ): All Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others:
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/sig testing /sig storage /assign
NOTE: the "local" provider is specifically hack/local-up- any other cluster that is not one of "gce" etc. should use the default provider / "skeleton" provider (I know the naming is bad, but it's sort of an API now)
/remove-sig testing