jiebi/CodeConvo huggingface dataset:
Code change:
if err != nil { return nil, "", err } <del> return aggregator.FilterSpecByPathsWithoutSideEffects(result, []string{"/apis/"}), etag, nil </del> <ins> group := specInfo.apiService.Spec.Group version := specInfo.apiService.Spec.Version return aggregator.FilterSpecByPathsWithoutSideEffects(result, []string{"/apis/" + group + "/" + version}), etag, nil </ins> }, cached.Result[*spec.Swagger]{Value: result, Etag: etag, Err: err}) specInfo.spec.Store(filteredResult) return err

Related discussion:
text: I tried to reproduce on my local setup with based on steps + , but I could still get a successful response when I executed . Then I reverted one of the step mentioned in to and waited until the new pod came alive and then did a apiserver restart and I was able to reproduce the same thing on as well i.e; I get the 503 response when I execute following command after the fact. After this point, even if i change it back to and restart the apiserver , the error is persistent. Let me know if this works for you to reproduce ? PS: Also, i want to call out that there is no issue with , its .
Thanks, alright I found a way to reproduce it consistently without restarting the apiserver. Still investigating the root cause. Thanks for the detailed reproduction and
Quick update: This issue was a combination of bugs both in k/k and prometheus adapter. I opened to mitigate on k/k side which should fix the issue. The issue was difficult to reproduce because it requires a very specific ordering of OpenAPI merging to trigger, which is why I suspect many people were able to reproduce after some restarts to kube-apiserver. Prometheus and metrics server serve differing OpenAPIs for the endpoint. Prometheus also doesn't seem to be registering discovery's OpenAPI properly (missing OpenAPI for but having the OpenAPI for whereas it is all present in metrics server. This inconsistency in how prometheus and metrics server handle the OpenAPI surfaced the merging issue.
a little wired question: why do we need to restart kube-apiserver to reproduce this problem?
Tried to reproduce the issue after cherry picking the fix and didn't run into below error. Fix is working as expected.
It can look like restarting prometheus-adapter "fixes" the issue? I am also curious about suggestion of removing the section from the config map works as a temporary workaround until released upstream (we use and cannot delete the apiregistration)? As I understand the fix for this issue is currently in
Code change:
fakeShutdownChan := make(chan bool) fakeDbus := &fakeDbus{currentInhibitDelay: tc.systemInhibitDelay, shutdownChan: fakeShutdownChan, overrideSystemInhibitDelay: tc.overrideSystemInhibitDelay} <ins> lock.Lock() </ins> systemDbus = func() (dbusInhibiter, error) { return fakeDbus, nil }

Related discussion:
text: /kind failing-test <!-- If this is a flake, please add: /kind flake --<!-- Please assign a SIG using: /sig SIG-NAME --
/sig node
I find the reason for it, but I don't know the solution is suitable, correct me if not.
/triage accepted /assign
Code change:
# --insecure-registry for local registries. DOCKER_OPTS=${DOCKER_OPTS:-""} <ins> # Extra options to set on the kube-proxy command line.  This is useful # for selecting the iptables proxy-mode, for example. KUBE_PROXY_EXTRA_OPTS=${KUBE_PROXY_EXTRA_OPTS:-""} </ins> # Optional: Install cluster DNS. ENABLE_CLUSTER_DNS="${KUBE_ENABLE_CLUSTER_DNS:-true}" # DNS_SERVER_IP must be a IP in SERVICE_CLUSTER_IP_RANGE

Related discussion:
text: There should be a way to select the iptables proxy-mode.
Are you interested in modifying ubuntu deploy scripts to use a user provided configure file?
This is fixed by and it will be merged in soon.
This fix breaks cluster/kube- while kubernetes provider is rather than ubuntu, as the following variables are only defined in ubuntu folder PROXYSETTING=${PROXYSETTING:-""} DEBUG=${DEBUG:-"false"} When I run KUBERNETES_PROVIDER=vagrant and run cluster/kube- It will fail with error "==master: [ERROR ] {'pid': 5871, 'retcode': 1, 'stderr': ' line 21: DEBUG: unbound variable', 'stdout': ''}" I will open a separated issue to discuss the solution.
I do not understand. I have not tried KUBERNETESPROVIDER=vagrant and run cluster/kube-, can you please clarify what is going wrong? This PR () affected only files in cluster/ubuntu/; are you saying that some of them are used when KUBERNETESPROVIDER=vagrant? This PR did not introduce PROXY_SETTING or DEBUG.
I tried running kube- with the Vagrant provider, but ran into a different (and much prior) problem; see --- any help would be appreciated.
OK, I see you opened issue for this and PR fixed it. Thanks. I will respond to the larger question under .
Code change:
"strings" "time" <ins> "github.com/ghodss/yaml" </ins> . "github.com/onsi/ginkgo" . "github.com/onsi/gomega"

Related discussion:
text: Is this a BUG REPORT or FEATURE REQUEST?: /kind bug What happened: The local-volume bootstrapper/provisioner configuration format changed in kubernetes-incubator/external-storage, and its related e2e tests will fail when the provisioner & bootstrapper images are updated to v2.0.0. What you expected to happen: I'd expect the e2e tests to continue to work properly with the new version of the provisioner and boostrapper. How to reproduce it (as minimally and precisely as possible): Update local-volume provisioner and boostrapper images to v2.0.0, and then the provisioner e2e test will fail.
/sig storage
/assign
Code change:
var _ = SIGDescribe("Network", func() { const ( <del> testDaemonHTTPPort    = 11301 testDaemonTCPPort     = 11302 timeoutSeconds        = 10 postFinTimeoutSeconds = 5 </del> <ins> testDaemonHTTPPort     = 11301 testDaemonTCPPort      = 11302 deadlineTimeoutSeconds = 10 postFinTimeoutSeconds  = 30 </ins> ) fr := framework.NewDefaultFramework("network")

Related discussion:
text: Which jobs are failing: kops-aws and many other jobs Which test(s) are failing: This isn't the actual test, but the text we're consistently seeing is Since when has it been failing: Testgrid links: (ref: ) Reason for failure: TBD Anything else we need to know: Very rarely tests pass when one of the tests passes: Seems related to , that was merged about the same time:
/sig cluster-lifecycle /sig aws
The label(s) cannot be applied, because the repository doesn't have them <detailsIn response to : /sig cluster-lifecycle /sig aws Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/priority critical-urgent this is merge blocking
/assign /cc /cc /cc
The PR we think we want to revert
is also failing, I don't think this is aws-specific
/cc
/assign
it failed with kind too :/ with the same message
Opened to revert
/unassign /sig network /sig testing /remove-sig cluster-lifecycle
/triage unresolved Comment when the issue is assessed and confirmed. I am a bot run by vllry. ‍
This may be the actual cause PR to address But we're going with revert first to be sure we've identified the offending PR
I think I've identified the root cause
Revert brought us back to green: Awaiting root cause fix
/remove-triage unresolved
/reopen merged but we're now seeing multiple tests fail since it merged Looking at it went red between these commits:
Reopened this issue. <detailsIn response to : /reopen merged but we're now seeing multiple tests fail since it merged Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
is an attempt to fix
reverting in
is the failure cluster I suspect will go down once the revert goes in
/close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"
Closing this issue.
Code change:
var _ = SIGDescribe("Network", func() { const ( <del> testDaemonHTTPPort    = 11301 testDaemonTCPPort     = 11302 timeoutSeconds        = 10 postFinTimeoutSeconds = 5 </del> <ins> testDaemonHTTPPort     = 11301 testDaemonTCPPort      = 11302 deadlineTimeoutSeconds = 10 postFinTimeoutSeconds  = 30 </ins> ) fr := framework.NewDefaultFramework("network")

Related discussion:
text: <detailsIn response to : /close Failure cluster has gone down, calling this done <img width="998" alt="Screen Shot 2020-01-10 at 9 56 17 AM" src="https://user-"</details
Code change:
"k8s.io/kubernetes/pkg/apis/extensions" ) <del> const ( // Annotation key used to identify mirror pods. mirrorAnnotationKey = "kubernetes.io/config.mirror" // Value used to identify mirror pods from pre-v1.1 kubelet. mirrorAnnotationValue_1_0 = "mirror" ) </del> // This is a "fast-path" that avoids reflection for common types. It focuses on the objects that are // converted the most in the cluster. // TODO: generate one of these for every external API group - this is to prove the impact

Related discussion:
text: Nodes can't recognize the static pods that they create possibly because the defaulting is different. always returns false. Here is a json diff of two pod specs before and after they are Semenatic.DeepEqual'd where before is created pod and after is what kubelet things the created pod should be. ! Thus: p0 for triage
I think the solution is to not apply defaults to mirror pods in the API Server.
Yes,we have to add specific code to API Server to detect the mirror pod, then not apply defaults.
The right fix should be , which we didn't get to before v1.0. I will start on this and at least get it in for v1.1 to prevent future complications. Since we cannot update the node/kubelet, options are: suggests to disable applying defaults to mirror pods as a temporary fix. IIRC, was against treating mirror pod differently at the apiserver. Since this is only a workaround, perhaps we can relax the constraint? a script to update the pod manifest files.
Disabling defaults has a number of problems: are version-specific. Conversion may fail if defaults are not applied. may fail if defaults are not applied. using defaulted fields assumes default values have been applied. defaulting pass is not the only place where values can be set, as mentioned in . For instance, if the user were to change the kube-system LimitRange, that would cause the same problem. This is another case where would have been useful. Which gives me another idea. For mirror pods, we could capture the entire metadata+spec provided by the user as an annotation, as we do in the client for kubectl apply, and then we could return that in GET. PUT may not work. We'd have to resolve the size concerns raised by . How serious of a problem is this delete-recreate loop?
Ok, we are going with approach proposed in . I am closing this issue as dup of
"entire metadata+spec" in plain text? It'd be hard for kubelet to diff using this. It'd be better to let kubelet store hash of its own choosing so that it can easily compare when there is any change.
Even with , we can't fix v1.0 kubelet.
I agree with
Yes, I were confused.
Code change:
"k8s.io/kubernetes/pkg/apis/extensions" ) <del> const ( // Annotation key used to identify mirror pods. mirrorAnnotationKey = "kubernetes.io/config.mirror" // Value used to identify mirror pods from pre-v1.1 kubelet. mirrorAnnotationValue_1_0 = "mirror" ) </del> // This is a "fast-path" that avoids reflection for common types. It focuses on the objects that are // converted the most in the cluster. // TODO: generate one of these for every external API group - this is to prove the impact

Related discussion:
text: The comparison is in Kubelet, and we cannot patch 1.0 kubelet in this case.
There are no serious issues with this bug: static pods are running properly, but corresponding mirror pods coming and gone. cc/ Can we treat this as part of GKE cluster upgrade process? Can we upgrade GKE master nodes, then ask the users to upgrade their fluentd yaml file in their cluster? For OSS kubernetes users, most likely they won't run into this issue. But we should document this as known issue for 1.1.
Or can we update the fluentd yaml files when updating the master?
cc/
This is a cc issue, really. But the fast answer is no, because "then ask the users to upgrade their fluentd yaml file in their cluster?" Historically our users are extremely slow to do maintenance.
Our current plan for this issue is just document as a known issue for 1.1 release.
In a 100 node cluster this is going to cause +100 writes +100 deletes per second. Is that extra load a concern?
This would be +100 writes +100 deletes every 10 seconds in a 100-node cluster. We are still discussing options to let user update their manifest files.
Ok, from the logs in the issue description, it looked more frequent but most of the deletes are failing with key not found
's easy to trigger a sync for all pods in kubelet. Any change in any pod would trigger a sync. In this case, the DELETE/ADD of the mirror pod could also trigger another sync. So, yes, it could be worse than 10s... We are looking at patching this in the apiserver, based on suggestion.
Yeah, my suggestion is fixing defaulting/conversion/validation as needed to make exceptions for mirror pods.
I'm not 100% sure it's needed, but if necessary we can turn the defaulting fix on in apiserver, and leave it off in scheduler, so that it gets the defaults it expects.
Yeah, it is way more frequent than 10s. I pointed this out to earlier. We had a meeting with and looks like: Providing a scripts to convert fluentd yaml file and asking the user to run is not a solution. Our users are very slow on such maintenance work. Piggyback such conversion work on master updating might not a solution for GKE clusters.
Code change:
"k8s.io/kubernetes/pkg/apis/extensions" ) <del> const ( // Annotation key used to identify mirror pods. mirrorAnnotationKey = "kubernetes.io/config.mirror" // Value used to identify mirror pods from pre-v1.1 kubelet. mirrorAnnotationValue_1_0 = "mirror" ) </del> // This is a "fast-path" that avoids reflection for common types. It focuses on the objects that are // converted the most in the cluster. // TODO: generate one of these for every external API group - this is to prove the impact

Related discussion:
text: We do have access to GKE nodes, and we don't do that. To obtain the permission to run script and sync a file requires more work from PM side. This is a API backward compatibility issue anyway. Given today's version management policy, we should handle such compatibility issue. I checked our default code: request diff was introduced by TerminationGracePeriodSeconds was introduced along with graceful termination support. There is no conversion for both fields There is no validation checking both fields today I think it is safe to disable such defaulting for mirror pods shouldn't cause any issue for 1.1 release. Sending PR shortly.
In order to not break the v1.1+ kubelet (which applies newer defaults), we should only skip defaulting for pods from pre-v1.1 kubelets. An easy way to do this is to reuse , which changes the mirror key annotation value from "mirror" to the hash of the pod manifest file. In short, if the key exists, it's a mirror pod. If the value is "mirror", it is sent by a pre-v1.1 kubelet. Otherwise, it's sent by a v1.1+ kubelet. Note that we will also need to maintain the special defaulting case for the future releases until we drop support for v1.0 nodes.
Just this morning, we discussed providing a script/tool to update static pod configs for people with large clusters. Why have we changed that plan? Any hack in apiserver is going to be confusing and fragile. For instance, a LimitRange installed for kube-system would defeat the hack. If we were to add a hack in apiserver, we'd need to remove it in 1.2.
we talked to and the conclusion was that node update may not be feasible as . I agree that any apiserver fix is hacky and fragile, but it seems the most viable option right now. We would need to carry this fix in apiserver as long as GKE supports v1.N master + v1.0 node, unless we can make updating the configs part of the cluster upgrade process.
Ah. These fields are problems because they existed in 1.0, but weren't used nor populated. Entirely new fields are silently dropped during decoding.
Code change:
"k8s.io/kubernetes/pkg/apis/extensions" ) <del> const ( // Annotation key used to identify mirror pods. mirrorAnnotationKey = "kubernetes.io/config.mirror" // Value used to identify mirror pods from pre-v1.1 kubelet. mirrorAnnotationValue_1_0 = "mirror" ) </del> // This is a "fast-path" that avoids reflection for common types. It focuses on the objects that are // converted the most in the cluster. // TODO: generate one of these for every external API group - this is to prove the impact

Related discussion:
text: We should cherrypick the solution into 1.0.7, since it appears there will be one.
That is, we should cherrypick into the 1.0 branch.
Ref.
If the hack is going to remain indefinitely, I'm not comfortable circumventing defaulting, conversion, and validation.
Just not performing the defaulting for these specific fields on just mirror pods seems safer.
In particular, I could easily imagine a new version of the pod API in 1.2 or 1.3, which will require defaulting and conversion to be applied, in general.
We'll be discussing this soon after 1.1 lands, I expect. My proposal would be for v1.N masters to support v1.N-1 and v1.N-2 (and maybe v1.N-3) nodes. That gives users 6-9 months to lag their nodes behind their masters. But it warrants much further discussion. This isn't a viable solution either, because we can't depend on users updating their nodes to the latest 1.0 rev. We expect arbitrary 1.0 nodes to work, including 1.0.6 all the way back to 1.0.1, per our current .
As Ike noted, cherrypicking into 1.0 kubelets is not feasible for solving this, because the whole reason for the problem is that people aren't updating their nodes. The conversion fix in is much preferable to the defaulting fix: it's less brittle, and components that need the defaulting (e.g. scheduler) will see the defaulting (because they run the conversion stack themselves), but on the wire the mirror pods from old kubelets will remain undefaulted.
I didn't say cherrypicking into 1.0 would solve the problem for everything. The behavior in 1.0 is pretty broken, even when not considering upgrade. We have a fix. We should cherrypick it into 1.0.
I don't see how different components would see different values for these fields, assuming they are all updated to the new code. As discussed offline, the scheduler and node controller need to handle cases of empty requests and pod termination for 1.0 kubelets, anyway, so I think the conversion approach is fine. Please put the explanation in code comments, though, not just in issue/PR comments.
Code change:
"k8s.io/kubernetes/pkg/apis/extensions" ) <del> const ( // Annotation key used to identify mirror pods. mirrorAnnotationKey = "kubernetes.io/config.mirror" // Value used to identify mirror pods from pre-v1.1 kubelet. mirrorAnnotationValue_1_0 = "mirror" ) </del> // This is a "fast-path" that avoids reflection for common types. It focuses on the objects that are // converted the most in the cluster. // TODO: generate one of these for every external API group - this is to prove the impact

Related discussion:
text: Since the conversion stack does the defaulting at the end and runs in every component, and scheduler, node-controller, etc all consume api.Pod objects (instead of v1.Pod objects), they will have a view of the object with the fields defaulted. The conversion based fix essentially retains the defaulting in *api.Pod objects, but undoes it in any serialized representation of the object. Over the wire the defaulting will not be baked in to the object. Older clients that have not been recompiled (including these kubelets) would see a version of the object without the fields being defaulted.
Defaulting is run first:
Oh, you're right, I somehow had that backwards in my head. Sorry, everyone. I think that tilts me back in favor of . This means that defaulting is part of the versioned structs and therefore changing the defaulting is an API breaking change, which we claimed we wouldn't do. Can we change our minds about the defaulting? How about moving this particular defaulting into conversion functions, which is the intended way of doing something like this?
In , we only reset the defaults when converting api.Pod to v1.Pod. Shouldn't all components still see the defaulting values? Agreed that this is an API breaking change. Perhaps we should not change the defaults at all? mentioned in that scheduler already deals with empty requests. It's true in , but seems untrue in . Maybe this should be fixed. As for , we can also ask kubelet to apply a default value if we truly on to have the 30 second dfault.
resources.requests and terminationGracePeriodSeconds are effectively newly fields, so setting defaults is not a breaking change for most clients. If Kubelet were comparing the pre-decoded pods returned by apisever, it would barf on every new field we've . It's an accident of the implementation that it broke on exactly these fields.
Actually, neither priorities nor predicates takes values from limits any longer. That code is rely on the defaulting pass to copy limits to requests in the case that the requests field is absent. We would need to change that, yes, if we were to not set requests for the 1.0 mirror pods.
This Kubelet behavior was known to be broken back in May: What we're trying to do now is figure out a workaround for this broken behavior.
Code change:
"k8s.io/kubernetes/pkg/apis/extensions" ) <del> const ( // Annotation key used to identify mirror pods. mirrorAnnotationKey = "kubernetes.io/config.mirror" // Value used to identify mirror pods from pre-v1.1 kubelet. mirrorAnnotationValue_1_0 = "mirror" ) </del> // This is a "fast-path" that avoids reflection for common types. It focuses on the objects that are // converted the most in the cluster. // TODO: generate one of these for every external API group - this is to prove the impact

Related discussion:
text: My recommended solution is: not set resources.requests and terminationGracePeriodSeconds to non-nil values for just 1.0 mirror pods. All other fields and all other pods, including 1.1 mirror pods, should have default values set normally. Doing this in the defaulting pass makes the most sense, assuming that it's straightforward to do given the structure of that pass. the scheduler fit and priority functions to use limits for pods where requests are entirely absent. that the new code in the node controller to handle termination of pods on 1.0 kubelets deals with unset terminationGracePeriodSeconds appropriately. a kind/workaround-removal issue to remove these hacks once we don't need to worry about 1.0 kubelets.
Just for comparison, an alternative: this default setting from the defaulting function. it to the conversion function instead. mirror pods from 1.0 kubelets (MPF1K), store the original value in an annotation. MPF1K, when converting to v1, restore the original value & remove the annotation. No changes needed anywhere else in the system, maintenance cost is very low, we can carry this as long as needed.
I don't see how that keeps the default values off the wire.
Actually, I think I see. Yes, I think that could work.
The defaults are on the v1-internal transition, and removed on the internal-v1 transition.
(3) is converting v1 to IR, and (4) is converting IR to v1, either for storing to etcd or to communicate to another component, such as Kubelet.
I'm saying stop using the default setting code for setting these "breaking" defaults.
Right.
I don't think we want to do this for pods in general, though, just for MPF1K
So I think we can actually just add a few more changes to PR and it should work. (Just move the defaulting.)
Yes, I agree. Just for MPF1K. We can easily broaden it if needed.
SGTM
We're going to want to do something similar for new fields to alpha API versions that we don't want to expose in the stable, serialized version of the API.
So if this works, we should add it to
Yes. Changing defaults is a breaking change and therefore needs to happen in conversion.