jiebi/CodeConvo huggingface dataset:
Change:
Request Headers: {% for i, key in ipairs(keys) do %} <del> {{key}}={{headers[key]}} </del> <ins> {% local val = headers[key] %} {% if type(val) == "table" then %} {% for i = 1,#val do %} {{key}}={{val[i]}} {% end %} {% else %} {{key}}={{val}} {% end %} </ins> {% end %} Request Body:

Discussion:
/kind bug Hello, actually this is a bug about not about k8s. I use to observe request headers and found if the request have multiple same headers echoserver will give a wrong result: result: Expected result:
/sig testing
any chance you know some lua?
I never used lua before, but I'm willing to try to fix this issue.
Change:
# keep request = limit to keep this container in guaranteed class limits: cpu: 10m <del> memory: 20Mi </del> <ins> memory: 50Mi </ins> requests: cpu: 10m <del> memory: 20Mi </del> <ins> # Note that this container shouldn't really need 50Mi of memory. The # limits are set higher than expected pending investigation on #29688. # The extra memory was stolen from the kubedns container to keep the # net memory requested by the pod constant. memory: 50Mi </ins> args: - -cmd=nslookup kubernetes.default.svc.{{ pillar['dns_domain'] }} 127.0.0.1 >/dev/null && nslookup kubernetes.default.svc.{{ pillar['dns_domain'] }} 127.0.0.1:10053 >/dev/null - -port=8080

Discussion:
Observed weird memory spikes for both the dns healthz and kube-dns pods in production. Kube-dns didn't oom because there's enough headroom, but healthz did. I can't explain the spikes yet. I tried a bunch of different repros: no , bad nslookup command, sharing the node with cpu hogs etc, but was unable to trigger it. The memory grows, but stays under 10Mi. As a short term mitigation, we stole some ram from kube-dns, because it was always comfortably within limits. Related b/
can be closed -- healthz has been removed
-- close issue?
Change:
continue } resp.Body.Close() <ins> if e, a := item.namespace, simpleStorage.requestedResourceNamespace; e != a { t.Errorf("%v: expected %v, got %v", item.rawQuery, e, a) } </ins> if e, a := item.resourceVersion, simpleStorage.requestedResourceVersion; e != a { t.Errorf("%v: expected %v, got %v", item.rawQuery, e, a) }

Discussion:
is always using the cross namespace context. This needs to be fixed to support scoped watches.
Change:
proxyStream(response.ResponseWriter, request.Request, url) } <ins> // trimURLPath trims a URL path. // For paths in the format of "/metrics/xxx", "metrics/xxx" is returned; // For all other paths, the first part of the path is returned. func trimURLPath(path string) string { parts := strings.SplitN(strings.TrimPrefix(path, "/"), "/", 3) if len(parts) == 0 { return path } if parts[0] == "metrics" && len(parts) > 1 { return fmt.Sprintf("%s/%s", parts[0], parts[1]) } return parts[0] } // isLongRunningRequest determines whether the request is long-running or not. func isLongRunningRequest(path string) bool { longRunningRequestPaths := []string{"exec", "attach", "portforward", "debug"} for _, p := range longRunningRequestPaths { if p == path { return true } } return false } </ins> // ServeHTTP responds to HTTP requests on the Kubelet. func (s *Server) ServeHTTP(w http.ResponseWriter, req *http.Request) { defer httplog.NewLogged(req, &w).StacktraceWhen(

Discussion:
<!-- Please only use this template for submitting enhancement requests --What would you like to be : 1) add metrics into kubelet to monitor ; 2) add throttling to . Why is this needed: Kubelet provides lots of its functionalities through its http server, and currently there is no server-side monitoring and throttling mechanisms. Adding metrics into kubelet to monitor its http server will improve its observability and debuggability; adding throttling will prevent kubelet being overloaded from serving too many http requests.
/sig node
Change:
<del> id: blog-service </del> <ins> apiVersion: v1beta3 </ins> kind: Service <del> apiVersion: v1beta1 port: 80 containerPort: 80 selector: name: blog createExternalLoadBalancer: true </del> <ins> metadata: name: blog-service spec: createExternalLoadBalancer: true ports: - port: 80 selector: name: blog </ins>

Discussion:
This causes . In contrib/git-sync demo, the container voluntarily exits and gets recreated to "periodically" sync the git repo, but the pod is then reported as not ready, The end result is that the service targets at this pod does not get endpoints and thus not working. Thanks to for helping with the debugging.
The container was not running when kubelet reported the status, hence it was considered not ready. Because the containers in a pod share the same fate, the pod was not perceived as not ready. Having short-lived containers as a service endpoint may encounter problems like this. I am not really convinced that this is kubelet's problem. It sounds like should be a cron job...
Forgot to cc
Is it alright if I modify git-sync to make it run a for-loop?
sure, but it would still be nice to able to have a way to have background containers restarting periodically in a pod w/o failing the service.
, will it be useful if we let the user to config which containers will be considered by kubelet when it reports the readiness of the pod?
I am not sure kubelet restart is the right mechanism to run periodic tasks, and in this case, recreating the container isn't particularly efficient. FYI, there is a discussion for adding a controller for cron jobs in .
Thanks Let's forget about letting kubelet restart periodic tasks. In general, if some containers are not critical to a service, ignoring the status of these containers and keeping the service alive will increase the availability of the service.
we'd like to have sidecar containers that aren't critical to a pod's overall functioning. i.e. a sidecar which can crash or become unavailable without pulling the pod from the load balancer. In our case it is a sidecar which delivers some configuration to disk from a notoriously flakey service. We'd like the pod to stay up, the conf it needs is on disk. Should I file a separate issue for this?
Change:
// Update replaces an existing event. Returns the copy of the event the server returns, or an error. func (c *FakeEvents) UpdateWithEventNamespace(event *v1.Event) (*v1.Event, error) { <del> action := core.NewRootUpdateAction(eventsResource, event) </del> <ins> var action core.UpdateActionImpl </ins> if c.ns != "" { action = core.NewUpdateAction(eventsResource, c.ns, event) <ins> } else { action = core.NewUpdateAction(eventsResource, event.GetNamespace(), event) </ins> } obj, err := c.Fake.Invokes(action, event) if obj == nil {

Discussion:
Opening this issue based on what I ran into on with namespace as follows While unit testing the controller, we create a fake client and setup the controller to run using the client. This ends up creating a having the namespace set to When the call eventually reaches The action gets associated with a namespace only if the object has a namespace associated with it. Which in this will be Which will result in the error in to manage the event like does. In case if the namespace is set to Above is a test case that can easily simulate the issue in question. Referring back to the test under PR NA <details</details<details</details<details</details<details</details<details</details<details</details
/sig api-machinery
/help /triage accepted
This request has been marked as needing help from a contributor. Please ensure that the issue body includes answers to the following questions: Why are we solving this issue? To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points? Does this issue have zero to low barrier of entry? How can the assignee reach out to you for help? For more details on the requirements of such an issue, please see and ensure that they are met. If this request no longer meets these requirements, the label can be removed by commenting with the command. <detailsIn response to : /help /triage accepted Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/assign
Change:
test: "Pod with an Affinity matches node's labels but the PodSpec.NodeSelector(the old thing that we are deprecating) " + "is not satisfied, won't schedule onto the node", }, <ins> { pod: &v1.Pod{ Spec: v1.PodSpec{ Affinity: &v1.Affinity{ NodeAffinity: &v1.NodeAffinity{ RequiredDuringSchedulingIgnoredDuringExecution: &v1.NodeSelector{ NodeSelectorTerms: []v1.NodeSelectorTerm{ { MatchExpressions: []v1.NodeSelectorRequirement{ { Key:      "foo", Operator: v1.NodeSelectorOpNotIn, Values:   []string{"invalid value: ___@#$%^"}, }, }, }, }, }, }, }, }, }, labels: map[string]string{ "foo": "bar", }, fits: false, test: "Pod with an invalid value in Affinity term won't be scheduled onto the node", }, </ins> } expectedFailureReasons := []algorithm.PredicateFailureReason{ErrNodeSelectorNotMatch}

Discussion:
SchedulerPredicates [Serial] validates that a pod with an invalid NodeAffinity is rejected Has been failing consistently since 4/17. Here are the /kind bug /priority failing-test /priority important-soon /sig scheduling /milestone 1.11 cc /assign for triage.
The provided milestone is not valid for this repository. Milestones in this repository: [, , , , , , , , , , ] Use to clear the milestone. <detailsIn response to : ## Failing Job SchedulerPredicates [Serial] validates that a pod with an invalid NodeAffinity is rejected /priority failing-test /priority important-soon /sig scheduling /milestone 1.11 Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/milestone v1.11
/status approved for milestone
Also fails as part of upgrade job as seen below
Also fails as part of following jobs https://k8s- https://k8s- https://k8s- https://k8s-
xref
/assign
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 6 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Seems like this is failing in (just with slightly different versions)
Change:
// Can't check exact strings due to terminal control commands (colors) requiredItems := []string{"Kubernetes master", "is running at"} if framework.ProviderIs("gce", "gke") { <del> requiredItems = append(requiredItems, "KubeDNS", "Heapster") </del> <ins> requiredItems = append(requiredItems, "Heapster") </ins> } for _, item := range requiredItems { if !strings.Contains(output, item) {

Discussion:
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . --Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: [sig-cli] Kubectl client [] Kubectl cluster-info should check if Kubernetes master services is included in cluster-info [Conformance] 10s e2e test "gci-gce" is running in 2 versions: with kube-dns as DNS Service : https://k8s- with CoreDNS as DNS Service : https://k8s- In CoreDNS use case the test is failing What you expected to happen: This e2e tests should not depend on what DNS service is running as default in K8s. CoreDNS is currently BETA, will move to GA soon (after v1.10 though) See : How to reproduce it (as minimally and precisely as possible): look at logs of dashboard : https://k8s- go run -v --test --test_args='--[sig-cli]sKubectlsclients[k8s.io]sKubectlscluster-infosshouldschecksifsKubernetessmastersservicessissincludedsinscluster-infoss[Conformance]$' Anything else we need to know?: Environment: Kubernetes version (use ): branch master Cloud provider or hardware configuration: GCE Install tools: deployment intra-test Others: see https://k8s-
/sig cli
Change:
} func (plugin *fcPlugin) ConstructVolumeSpec(volumeName, mountPath string) (*volume.Spec, error) { <del> fcVolume := &v1.Volume{ Name: volumeName, VolumeSource: v1.VolumeSource{ FC: &v1.FCVolumeSource{}, }, </del> <ins> // Find globalPDPath from pod volume directory(mountPath) // examples: //   mountPath:     pods/{podUid}/volumes/kubernetes.io~fc/{volumeName} //   globalPDPath : plugins/kubernetes.io/fc/50060e801049cfd1-lun-0 var globalPDPath string mounter := plugin.host.GetMounter(plugin.GetPluginName()) paths, err := mount.GetMountRefs(mounter, mountPath) if err != nil { return nil, err } for _, path := range paths { if strings.Contains(path, plugin.host.GetPluginDir(fcPluginName)) { globalPDPath = path break } } // Couldn't fetch globalPDPath if len(globalPDPath) == 0 { return nil, fmt.Errorf("couldn't fetch globalPDPath. failed to obtain volume spec") } arr := strings.Split(globalPDPath, "/") if len(arr) < 1 { return nil, fmt.Errorf("failed to retrieve volume plugin information from globalPDPath: %v", globalPDPath) } volumeInfo := arr[len(arr)-1] // Create volume from wwn+lun or wwid var fcVolume *v1.Volume if strings.Contains(volumeInfo, "-lun-") { wwnLun := strings.Split(volumeInfo, "-lun-") if len(wwnLun) < 2 { return nil, fmt.Errorf("failed to retrieve TargetWWN and Lun. volumeInfo is invalid: %v", volumeInfo) } lun, err := strconv.Atoi(wwnLun[1]) if err != nil { return nil, err } lun32 := int32(lun) fcVolume = &v1.Volume{ Name: volumeName, VolumeSource: v1.VolumeSource{ FC: &v1.FCVolumeSource{TargetWWNs: []string{wwnLun[0]}, Lun: &lun32}, }, } glog.V(5).Infof("ConstructVolumeSpec: TargetWWNs: %v, Lun: %v", fcVolume.VolumeSource.FC.TargetWWNs, *fcVolume.VolumeSource.FC.Lun) } else { fcVolume = &v1.Volume{ Name: volumeName, VolumeSource: v1.VolumeSource{ FC: &v1.FCVolumeSource{WWIDs: []string{volumeInfo}}, }, } glog.V(5).Infof("ConstructVolumeSpec: WWIDs: %v", fcVolume.VolumeSource.FC.WWIDs) </ins> } return volume.NewSpecFromVolume(fcVolume), nil }

Discussion:
Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: Currently, FC plugin returns volume name and empty FCVolumeSource during ConstrutVolumeSpec during . However, in order to recreate newMounter from volumeSpec, volume information are lacking. As a result, filesystem volume's reconstruction always fails. What you expected to happen: Filesystem volume's reconstruction works properly, then a volume is unmounted and detached from kubelet node. How to reproduce it (as minimally and precisely as possible): (1) Create filesystem volume using FC plugin (2) Restart kubelet service which a pod with FC volume is running Anything else we need to know?: cc Environment: Kubernetes version (use ):Install tools: local-cluster-
/assign
/sig storage
Change:
fmt.Errorf("Fake error"), true, 0, 3, 0, 0, 0, 1, 3, 0, 0, nil, "", }, <del> "failed pod": { </del> <ins> "failed + succeed pods: reset backoff delay": { </ins> 2, 5, 6, false, 0, <del> fmt.Errorf("Fake error"), false, 0, 1, 1, 1, </del> <ins> fmt.Errorf("Fake error"), true, 0, 1, 1, 1, </ins> 1, 0, 1, 1, 1, nil, "", }, <ins> "only new failed pod": { 2, 5, 6, false, 0, fmt.Errorf("Fake error"), false, 0, 1, 0, 1, 1, 0, 1, 0, 1, nil, "", }, </ins> "job finish": { 2, 5, 6, false, 0, nil, true, 0, 0, 5, 0,

Discussion:
This issue was born out of the discussion from and In short: Job controller has an exponential backoff in place when failures happen to a job. As long as the parallelism is one, it's clear when a failure or success happens, the latter clears the backoff. The problem is when parallelism 1, which can lead to situation where the controller can observe successful and failed pod at the same time and won't clear the backoff leading to further retries and longer wait times (see for extensive details about the problem in e2e) Possible solutions: everything as is both success and failure are observed clear the backoff Pros and cons: Ad. 1. Aside from () I wonder if users got affected by it. Does it make sense to fix it at all? Ad. 2. This approach make sense in some situation, but the question arise, should that be controlled by a user. What if I have 9 failures and 1 success (parallelism = 10), should I still clear the backoff? Maybe we should introduce user controller field to set the failure rate in this case.
/cc
(jumping here 'cause forwarded me this). I personally think that the right approach is to reset the back-off as soon we have a successful pod run to complete. Back-off was introduced, at least in our case, to reduce the impact of bad configured PodSpec in Jobs. If in your run you do have some success, it means you've a good config and you just keep trying (not that different that what we did before ). I don't think it's worth asking user finer tune config. is already complex enough. My 2 cents.
cc
I agree with any success should immediately clear the backoff flag, since it shows the job is configured properly and the problem is only temporary. I'll assign it to myself, but anyone is open to submit a PR fixing this.
Hi I started to implement the pod success check in order to clear the backoff delay in case of at least one of the pods succeed. let me know if you are fine with this approach, and then I will go further with unit-tests and end2end tests.
is the rest of the work on this issue getting done for 1.10? Please update, and add approved-for-milestone if this is on track for 1.10.
I updated my change with missing unit-tests and created the PR:
Change:
record_command run_wait_tests <ins> #################### # kubectl debug    # #################### if kube::test::if_supports_resource "${pods}" ; then record_command run_kubectl_debug_pod_tests fi if kube::test::if_supports_resource "${nodes}" ; then record_command run_kubectl_debug_node_tests fi </ins> cleanup_tests }

Discussion:
What would you like to be : Add integration tests for Why is this needed: This is part of the kubernetes/enhancements /sig cli /priority important-soon
/assign
/triage accepted
Change:
zoneSizeKey             = "zone_size" zoneNoUnhealthyNodesKey = "unhealthy_nodes_in_zone" evictionsNumberKey      = "evictions_number" <ins> evictionsTotalKey       = "evictions_total" </ins> ) var (

Discussion:
I would like to propose the graduation of the metric from alpha to stable. The metric has been stable for 2 years now.
/sig api-machinery /sig instrumentation
I am willing to do all StabilityLevels from alpha to stable? Is that right?
/assign
- The request is to have the stability level for the metric graduate to stable. If there are other metrics you want to do at the same time that's a bonus, but not part of the request in this issue.
/sig node /remove-sig api-machinery /cc
To be promoted to stable, the metrics would need to be renamed from to in order to follow Prometheus best practices as well as sig-instrumentation naming guidelines.
/cc
/assign
/triage accepted
Change:
return host, nil } <ins> // GetAddressesByTypeAndFamily returns a list of addresses of the given addressType for the given node // and filtered by IPFamily func GetAddressesByTypeAndFamily(node *v1.Node, addressType v1.NodeAddressType, family v1.IPFamily) (ips []string) { for _, nodeAddress := range node.Status.Addresses { if nodeAddress.Type != addressType { continue } if nodeAddress.Address == "" { continue } if family == v1.IPv6Protocol && netutil.IsIPv6String(nodeAddress.Address) { ips = append(ips, nodeAddress.Address) } if family == v1.IPv4Protocol && !netutil.IsIPv6String(nodeAddress.Address) { ips = append(ips, nodeAddress.Address) } } return } </ins> // GetAddresses returns a list of addresses of the given addressType for the given node func GetAddresses(node *v1.Node, addressType v1.NodeAddressType) (ips []string) { for j := range node.Status.Addresses {

Discussion:
<!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: sig-network-kind-ipvs,dual,master sig-network-kind-ipvs,IPv6,master Which test(s) are failing: [x] KubeProxy should set TCP CLOSE_WAIT timeout [x] Conntrack should be able to preserve UDP traffic when server pod cycles for a NodePort Service [x] KubeProxy should resorve connection reset issue (tracking in ) Since when has it been failing: Oct 28 Testgrid link: sig-network-kind-ipvs,dual,master  Reason for failure: Both appear to be failing on missing conntrack entries ex. Anything else we need to know:
/sig network /cc /assign
I'm spinning up a kind cluster with ipv6+ipvs to see if I can replicate this test locally
/triage accepted
heh, the job was not running it, it started to run when we changed the tag I meant this test and the PR where we renamed the kube-proxy test
this one has an issue already open, and I have a PR for fixing it
reached out to me on Slack and we identified as the high priority. I have tested this locally and it fails on both IPTables and IPVS kind clusters locally. I'm digging through the cluster logs now.
I' ve logging to conntrack with level v4 recently just for these problems, try to bump your kube-proxy logging to v4
seems that there are less test failures on the IPv6 jobs, let's keep watching them :eyes: However, the dual-stack jobs are legit failures, basically because on dualstack clusters, the node has 2 internal IPs, and those tests use the internal IP for asserting some operations, and the problem is that they are picking the internalIP of the wrong IP family, hence the test fails The solution should be to add the InernalIp correspoding to the cluster primary IP familty, you can obtain it with but it seems that we have to modify the function or add a new one that returns the nodes and the corresponding ips
Hey Thanks for all the help, will look into the areas of the code and try to bump the logging.
Change:
klog.V(5).Infof("Installing health checkers for (%v): %v", path, formatQuoted(checkerNames(checks...)...)) <del> mux.Handle(path, handleRootHealthz(checks...)) </del> <ins> mux.Handle(path, metrics.InstrumentHandlerFunc("GET", /* group = */ "", /* version = */ "", /* resource = */ "", /* subresource = */ path, /* scope = */ "", /* component = */ "", handleRootHealthz(checks...))) </ins> for _, check := range checks { mux.Handle(fmt.Sprintf("%s/%v", path, check.Name()), adaptCheckToHandler(check.Check)) }

Discussion:
What would you like to be : Currently the apiserver has request latencies metrics, which are a histogram of requests to the apiserver. However, these request metrics do not include healthz. While we have recently the ability to get back health check metrics from the healthz endpoint directly, we should also emit this data as a metric, so that we can monitor health check latencies. /assign /sig api-machinery instrumentation
GitHub didn't allow me to assign the following users: jktomer. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : What would you like to be : /sig api-machinery instrumentation Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
I noticed this is informally assigned. Has there been any progress? Is this a place I can fit in?
Yes, I'm working on a PR.
/assign
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-testing, kubernetes/test-infra and/or . /lifecycle stale
/remove-lifecycle stale