jiebi/CodeConvo huggingface dataset:
Code change:
}) It("should have all expected certs on the master", func() { <del> if testContext.Provider != "gce" && testContext.Provider != "gke" { </del> <ins> if !providerIs("gce", "gke") { </ins> By(fmt.Sprintf("Skipping MasterCerts test for cloud provider %s (only supported for gce and gke)", testContext.Provider)) return }

Related discussion:
text: I noticed while running e2e tests locally that some tests were apparently being incorrectly skipped: I checked, and this is also true for our Jenkins runs. Currently it looks like and have this bug, though only the former actually skips the test, I think. The issue appears to be that the code inside the top-level in these files is being run before the testContext struct is filled out (and before is even called), and so testContext.Provider is empty. I'm not immediately sure what the proper fix is; I'm also worried that there might be other things broken by the fact that these blocks are called before initialization is complete. We also probably investigate if there is a better way to mark these tests as skipped; right now, there's no indication in the test report (, Jenkins test report, etc) that these tests were skipped.
I merged and which tried to disable these tests for non-{gke,gce} but apparently didn't follow-up and verify that they were still running on our platforms. /cc
Seems easy enough to fix. I'll have a PR out to fix this before Monday.
Code change:
if ds.Status.DesiredNumberScheduled == desiredNumberScheduled && ds.Status.CurrentNumberScheduled == currentNumberScheduled && ds.Status.NumberMisscheduled == numberMisscheduled { return nil } var updateErr, getErr error for i := 0; i <= StatusUpdateRetries; i++ { ds.Status.DesiredNumberScheduled = desiredNumberScheduled

Related discussion:
text: The test is in flaky suite and no one seems to care. We need an owner for this test. cc Last failure log:
do you think you can take a look at this?
This is happening because of a bad assumption. The assumption is: if the reflector's store is consistent with etcd, the store given back to clients of the framework.Controller is too. I'd really like a general fix applicable to all controllers for this problem, but fixing it just for the RCs is pretty easy. In more detail: This function reports when the reflector has listed from apiserver. But the store given to the reflector is the fifo of the framework.Controller. The store given back to the caller is still not consistent, and will not be till the framework.Controller puts all the objects from the fifo into the client state store (). The rc specific fix: Before starting the rc manager, seed the expectations store with an expectation like {rc:foo, adds: foo.Status.Replicas, deletes:0}. This will wait for status.Replicas number of ADD events till a timeout. The general fix: Somehow expose this through framework.Controller.HasSynced (clients don't care about the fifo anyway, shouldn't it just be an implementation detail?) ideas? cc'ing people who have written controllers that might be interested I don't think this (and generally anything flaky for 1 release) is a p0, I can fix it if you don't mind dealing with the flakes till I've cleared things off my plate. I'd be more than happy to review a fix.
Well, it's a P0 because test is flaky. E.g last run:
- did you have chance to look into it?
The "DaemonRestart" tests are still labeled at . I will send a PR to promote them.
are you still planning to send aPR to promote the DaemonRestart tests out of flaky? I think this is the right thing to do.
Ack; sorry, I lost this in the muck.
Code change:
ObjectMeta: metav1.ObjectMeta{ Name: string(kl.nodeName), Labels: map[string]string{ <del> metav1.LabelHostname:       kl.hostname, metav1.LabelOS:             goruntime.GOOS, metav1.LabelArch:           goruntime.GOARCH, metav1.LabelFluentdDsReady: "true", </del> <ins> metav1.LabelHostname: kl.hostname, metav1.LabelOS:       goruntime.GOOS, metav1.LabelArch:     goruntime.GOARCH, </ins> }, }, Spec: v1.NodeSpec{

Related discussion:
text: PR introduced an annotation for fluentd migration from static pod to the daemonset to avoid two instances of fluentd on a single node during the transition stage. I didn't read the pr carefully, but based on the code at , every node object has one label be default: "". I think this is wrong. 1) Not everyone use fluentd as the logging agent. 2) It is hard-coded in Kubelet code as part of kubeletnodestatus. No matter running fluentd as a static pod or a daemonset on the node, Kubelet shouldn't treat it different from other pods. The right approach is introducing an external controller to apply this cloud-provider / kubernetes vendor (i.e. GKE) specific label to the node. cc/
The string "fluentd" shouldn't appear anywhere in our code. Please remove it ASAP.
Absolutely agree with you, thanks for bringing this up! Is it possible to apply labels to the node as part of the node startup script/salt?
To clarify, I'm thinking about possible ways to provide a fix for that ASAP. Writing external controller is a best solution, but here we have a release coming and this label is intended to be only a temporary measure to perform upgrade from using fluentd as static pods to using daemon set.
There are many ways to achieve this outside the core system. But I believe GKE already has an existing mechanism to add this kind of labels to the node, and the consistency is important for GKE management. cc/
OK, thanks! However, I believe we should be sure not to break everyone's upgrade process, i.e. after master upgrade to 1.6 on any platform, there should be only one fluentd agent on each node. I propose changing the configure scripts and salt to set up needed labels in kubelet's flag (). WDYT?
It seems to me like upgrades could survive without the explicit dependency on fluentd: I'm not sure if upgrades call initialNode or not, my own ignorance ? If they do, and this is removed, can't an external controller just do the work of re--adding that label if its dropped?
thanks for raising this. Let's sync f2f
cc
WDYT about exposing kubelet version as a label and have a selector in fluentd ds config targeting version 1.6+?
Code change:
ObjectMeta: metav1.ObjectMeta{ Name: string(kl.nodeName), Labels: map[string]string{ <del> metav1.LabelHostname:       kl.hostname, metav1.LabelOS:             goruntime.GOOS, metav1.LabelArch:           goruntime.GOARCH, metav1.LabelFluentdDsReady: "true", </del> <ins> metav1.LabelHostname: kl.hostname, metav1.LabelOS:       goruntime.GOOS, metav1.LabelArch:     goruntime.GOARCH, </ins> }, }, Spec: v1.NodeSpec{

Related discussion:
text: and I talked about this issue a bit for migrating kube-proxy. One idea we came up with was that of running a controller that will identify kubelet versions and apply taints to nodes running static pods. The daemonset version would not tolerate that taint and that way we keep daemon set pods off of nodes running static pods. Have you considered that approach?
It doesn't differ from a controller applying labels. I'd like to avoid necessity of introducing such controller, since it's not clear what is it's responsibility (generic controller for applying labels on nodes based on some rules? a custom controller just for fluentd purposes). In GKE we can do it on GKE control plane side, but we need a story for open source.
Move applying labels to startup scripts/salt seems to be the least intrusive and easiest to implement way.
Actually what I proposed in is not an option since nodeSelector in PodSpec is old style selector.
I'm not a big fan of approach with salt, but it seems there is no better option.
Re: Instead of exposing Kubelet version as a label, why not using field selector. Kubelet version is part of NodeStatus already. I just simply don't want to have a duplicate info for the same object.
is it possible to schedule based on fieldselector? I'm only aware of using field selector while listing objects.
Also node-labels option is in alpha
But it's present since 1.2 What is the status of this flag? Can we use it for fluentd purposes or better avoid this?
Maybe it does make sense to implement a temporary ad-hoc controller? It's the easiest to deploy solution, just another in fluentd addon.
I know. It totally doesn't make sense to introduce anything custom. I'll use just the flag. It's rather hacky but unfortunately I can't see anything better.
Cool thing about controller is that you can deploy it into any cluster and it will work, with CA also. You don't have to set up your node labels manually then or restart the cluster.
There is a number of open question with having a new controller: it run on the master or a node? we guarantee that it will be always running? it have any resource requirements? I'd like to avoid discussing it right before the release.
Having a controller will help with kube-proxy transition too in the future.
Code change:
ObjectMeta: metav1.ObjectMeta{ Name: string(kl.nodeName), Labels: map[string]string{ <del> metav1.LabelHostname:       kl.hostname, metav1.LabelOS:             goruntime.GOOS, metav1.LabelArch:           goruntime.GOARCH, metav1.LabelFluentdDsReady: "true", </del> <ins> metav1.LabelHostname: kl.hostname, metav1.LabelOS:       goruntime.GOOS, metav1.LabelArch:     goruntime.GOARCH, </ins> }, }, Spec: v1.NodeSpec{

Related discussion:
text: , Piotr Szczesniak <:
is this a 1.6 release blocker?
I think yes. The PR is already sent and approved. I need to fix e2e tests because it doesn't work well with the test infra (e2e tests passes locally). EDIT: I'll do it hopefully today.
The PR is ready to be merged and lgtmed. Waiting for approvals.
Code change:
expectedPatchDataPattern: `{"status":{"$setElementOrder/conditions":[{"type":"currentType"}],"conditions":[{"lastProbeTime":"2020-05-13T01:01:01Z","message":"newMessage","reason":"newReason","type":"currentType"}]}}`, }, { <del> name: "Should make patch request if pod condition already exists and is identical but nominated node name is different", </del> <ins> name: "Should not make patch request if pod condition already exists and is identical and nominated node name is not set", </ins> currentPodConditions: []v1.PodCondition{ { Type:               "currentType",

Related discussion:
text: <!-- Please only use this template for submitting reports about continuously failing tests or jobs in Kubernetes CI --Which jobs are failing: Conformance - GCE - master (ci-kubernetes-gce-conformance-latest) Which test(s) are failing: [sig-scheduling] SchedulerPreemption [Serial] PreemptionExecutionPath runs ReplicaSets to verify preemption running path [Conformance]Changes Since when has it been failing: 2020-06-08 22:27 PDT Testgrid link: Reason for failure: Regression by Anything else we need to know: /sig scheduling /priority critical-urgent
cc/ /assign
Code change:
MUST match to expected used and total allowed resource quota count within namespace. release: v1.16 file: test/e2e/apimachinery/resource_quota.go <ins> - testname: ResourceQuota, manage lifecycle of a ResourceQuota codename: '[sig-api-machinery] ResourceQuota should manage the lifecycle of a ResourceQuota [Conformance]' description: Attempt to create a ResourceQuota for CPU and Memory quota limits. Creation MUST be successful. Attempt to list all namespaces with a label selector which MUST succeed. One list MUST be found. The ResourceQuota when patched MUST succeed. Given the patching of the ResourceQuota, the fields MUST equal the new values. It MUST succeed at deleting a collection of ResourceQuota via a label selector. release: v1.25 file: test/e2e/apimachinery/resource_quota.go </ins> - testname: ResourceQuota, quota scope, BestEffort and NotBestEffort scope codename: '[sig-api-machinery] ResourceQuota should verify ResourceQuota with best effort scope. [Conformance]'

Related discussion:
text: [x] test approval issue : [x] test pr : 7 July 2022 [X] two weeks soak end date : 21 July 2022 [X] test promotion pr : According to this APIsnoop query, there is still some remaining ResourceQuota endpoints which are untested. The Status endpoints will be addressed in another test. listCoreV1ResourceQuotaForAllNamespacespatchCoreV1NamespacedResourceQuotadeleteCoreV1CollectionNamespacedResourceQuota Using a number of existing e2e test practices a new has been created for three ResourceQuota endpoints. The e2e logs for this test are listed below. This query shows the endpoints hit within a short period of running the e2e test If a test with these calls gets merged, test coverage will go up by 3 points This test is also created with the goal of conformance promotion. /sig testing /sig architecture /area conformance
/triage accepted
/reopen
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
expectedErr:     true, }, { <del> desc:        "correct LUN and no error shall be returned if everything is good", vmList:      map[string]string{"vm1": "PowerState/Running"}, nodeName:    "vm1", existedDisk: compute.Disk{Name: to.StringPtr("disk-name"), DiskProperties: &compute.DiskProperties{Encryption: &compute.Encryption{DiskEncryptionSetID: &diskEncryptionSetID, Type: compute.EncryptionAtRestWithCustomerKey}}, Tags: testTags}, </del> <ins> desc:     "correct LUN and no error shall be returned if everything is good", vmList:   map[string]string{"vm1": "PowerState/Running"}, nodeName: "vm1", existedDisk: compute.Disk{Name: to.StringPtr("disk-name"), DiskProperties: &compute.DiskProperties{ Encryption: &compute.Encryption{DiskEncryptionSetID: &diskEncryptionSetID, Type: compute.EncryptionAtRestWithCustomerKey}, DiskSizeGB: to.Int32Ptr(4096), }, Tags: testTags}, </ins> expectedLun: 1, expectedErr: false, },

Related discussion:
text: <!-- Please use this template while reporting a bug and provide as much info as possible. Not doing so may result in your bug not being addressed in a timely manner. Thanks! If the matter is security related, please disclose it privately via --What happened: there is a disk resize scenario: In the beginning, it’s 3TiB with cachingMode=ReadOnly, and then disk detached, resized to 4TiB, then user find they could not attach that disk back to any agent node, I think that’s a bug we need to fix. What you expected to happen: How to reproduce it (as minimally and precisely as possible): Anything else we need to know?: Environment: Kubernetes version (use ): Cloud provider or hardware configuration: OS (e.g: ): Kernel (e.g. ): Install tools: Network plugin and version (if this is a network-related bug): Others: /kind bug /assign /priority important-soon /sig cloud-provider /area provider/azure
This issue is currently awaiting triage. If a SIG or subproject determines this is a relevant issue, they will accept it by applying the label and provide further guidance. The label can be by org members by writing in a comment. <detailsInstructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Code change:
sanitizedSpecVolID := kstrings.EscapeQualifiedNameForDisk(specVolID) return path.Join(host.GetVolumeDevicePluginDir(csiPluginName), sanitizedSpecVolID, "data") } <ins> // hasReadWriteOnce returns true if modes contains v1.ReadWriteOnce func hasReadWriteOnce(modes []api.PersistentVolumeAccessMode) bool { if modes == nil { return false } for _, mode := range modes { if mode == api.ReadWriteOnce { return true } } return false } </ins>

Related discussion:
text: <!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup is only intended for RWO volumes because it recursively changes the permissions of all the directories in the volume. This can cause problems for nfs-type volumes because multiple pods could access the same volume and have the permissions changed out from underneath. What you expected to happen: Determine whether or not the plugin supports fsgroup either via: access mode. This is still a bit ambiguous for nfs volume types, that can theoretically support rwo, especially since Kubernetes does not do any sort of access mode enforcement. a new csi capability
CC
Are you saying that CSI internal driver should check driver capability for RWO, if supported, only then apply value ? What would the be ?
Yes one possible way is to infer fsgroup is supported for all rwo volumes, which may not be true? Like a nfs volume technically supports rwo but not fsgroup. So the other option is to make it a new CSI capability. I'm not sure exactly sure how best to describe it though, maybe something about volume ownership management.
Will it work if is not set then don't chown the ownership? I would think that for shared file system types(nfs, glusterfs) - won't be set. For raw block storage too - I think we shouldn't have to do this.
what do you mean by ? Shouldnt have to solve this issue ?
I meant - which is obvious. I just meant that the entire mechanism of changing ownership of files should apply only to volume types with block storage filesystems. I am just wondering if we can get away with detecting fsType and not have to introduce new capability in CSI.
agreed, do not map well to current csi.Capabilities enums which makes it hard to determine RWO when mounting. Perhaps, instead of a new capability, we revisit the existing capabilities to see if there are any way RWO can be reliably inferred from returned capabilities.
Code change:
sanitizedSpecVolID := kstrings.EscapeQualifiedNameForDisk(specVolID) return path.Join(host.GetVolumeDevicePluginDir(csiPluginName), sanitizedSpecVolID, "data") } <ins> // hasReadWriteOnce returns true if modes contains v1.ReadWriteOnce func hasReadWriteOnce(modes []api.PersistentVolumeAccessMode) bool { if modes == nil { return false } for _, mode := range modes { if mode == api.ReadWriteOnce { return true } } return false } </ins>

Related discussion:
text: Currently, this is what the code is doing which we knew needed to be revisited:
Users may not fully understand what fsgroup is and set it anyway, or a default psp with some fsgroup may be defined in which it will automatically be set on all pods. The first case we can attribute to user error, but the second case may not have a good workaround and could cause permission issues on rwx volumes. So I think we do need to solve this.
Yes, agreed this has to be solved. As you pointed out user error or improperly configured PSP can set value that does not match the correct attribute, causing permission issues. Here is what I think can be done without new CSI capabilities: , then do not apply fsgroup then: , apply fsgroup , do not apply fsgroup to void stated issue then: , apply fsgroup I think that may help deduce when to apply fsgroup.
If mount reference count <= 1, apply fsgroup If mount reference count 1, do not apply fsgroup to void stated issue A RWM volume (say NFS) can be mounted on multiple nodes. Do you have reliable count of mounts across all nodes?
In addition, that would be a change in semantics to how we support rwx volumes today. If we don't have a new CSI capability, then I think the best course of action is to only apply fsgroup for RWO volumes. Some questions remain: Do all RWO volumes support fsgroup? RWX is a superset of RWO. So if a user requests a RWO volume, it could still be satisfied by a RWX volume type. Can we detect this and still not apply the fsgroup even though the volume capability will say RWO?
If there's not a reliable way to enumerate node mounts (thinking about it, for CSI external drivers, the answer is NO). Theres is no reliable way to have k8s autocorrect a bad fsgroup/PV.AccessMode combo. So we would have to rely on the config provided by user/admin. I have come to similar conclusion as -- mainly, apply fsgroup only to volumes with access modes. The other k8s specified access modes, , cannot be reliably applied.
Code change:
sanitizedSpecVolID := kstrings.EscapeQualifiedNameForDisk(specVolID) return path.Join(host.GetVolumeDevicePluginDir(csiPluginName), sanitizedSpecVolID, "data") } <ins> // hasReadWriteOnce returns true if modes contains v1.ReadWriteOnce func hasReadWriteOnce(modes []api.PersistentVolumeAccessMode) bool { if modes == nil { return false } for _, mode := range modes { if mode == api.ReadWriteOnce { return true } } return false } </ins>

Related discussion:
text: Even if we were to introduce additional plugin capabilities that the CO can query (from driver) to find out what modes can be applied, there would still be the possibility of user/admin mis-configuration. Probably not, Is this where you think an additional capability would help ( ) That is a good point. Can't think of a way to detect/guess RWX when RWO is specified, this without the driver providing that info. Can we adopt the rule until we decide what capabilities would look like ?
I will just try to summarize the conversation from Slack. We can safely assume that only applies to block storage type with file system on top of it. So for all - volumes that need to have called MUST have a file system() on them. Is that assumption correct? If it is - can we not use presence of field or we can query the local volume for fs type (like how we query selinux relabelling capability) and call only when - volume is RW and has a valid block storage file system type. Will this not solve our use case? EDIT: renamed to .
We'll need to make it clear that is only for block-based storage systems, and the field cannot be reused for some multi-protocol file server (ie, nfsv3, nfsv4, smb, etc). For that case, then the plugin will need to expose its own new storageclass parameter + volume attribute to specify protocol.
Can we not use to get fsType - and use that information for calling ? It may be problematic to rely entirely on present inside csi source.
the only problem with is that in CSI volume info is intentionally opaque to the CSI code inside k8s. All mount logic is delegated to the driver.
Instead of a new capability or trying to deduce how best to apply fsgroup, why not pass the to the driver and let it decide how to apply it (either as a attribute for the mount or top level request param)? Right now, all mount logic has been delegated to the external CSI driver except for applying permission. Thoughts ?
that is true but we can still inspect file system on the mounted path because by the time gets called volume is already mounted. This should be no different than how selinux support is determined. what you think?
is this approved for 1.12 milestone? Thank you for working on this,
still discussing how best to approach this.