jiebi/CodeConvo huggingface dataset:
Once e2e tests are running, the tests keep running even if I press and exit the process that controls the TTY. But the tests actually keep running in the background and they keep printing stuff on a shell prompt which has already returned and thereby clobbering the STDOUT
/assign
GitHub didn't allow me to assign the following users: oomichi. Note that only can be assigned. <detailsIn response to : /assign Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Solution: I will add one.
/cc
xref
Thanks for sending a PR so quickly!
[MILESTONENOTIFIER] Milestone Issue Needs Attention Action required: During code slush, issues in the milestone should be in progress. If this issue is not being actively worked on, please remove it from the milestone. If it is being worked on, please add the label so it can be tracked with other in-flight issues. Note: If this issue is not resolved or labeled as by Wed, Nov 22 it will be moved out of the v1.9 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
During OPNFV test execution we noticed the folder selinux context changed. e2e test log How the folder changed after the test executed. I expected the TC won't change the selinux context. The question here, should this sigstorageserial TC set work on an selinux enabled K8S cluster or not? This question valid for the another sigstorage TC set with several test case, where selinux block a lot of TC too. See above. No response_ <details</details<detailsOpenStack </details<details</details<details</details<details</details<details</details
/sig testing
/sig storage
The Kubernetes project currently lacks enough contributors to adequately respond to all issues. This bot triages un-triaged issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue as fresh with Close this issue with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
/assign
Reproducer with simple kubernetes/kubernetes artifacts:
/triage accepted
Forking from To summarize, here's what we observed: daemonset calls coming every minute from both fluentd-scaler and addon-manager (verified this by individually turning them on/off). Things we need to understand here: Is the daemonset object continuously toggling b/w 2 states? (we know here that it's RV is increasing continuously) If yes, what field(s) in the object are changing? IIRC value of some label/annotation (I think 'UpdatedPodsScheduled') is changing (probably related to 2 below) Also, why/should the fluentd-scaler send any api request if the resources are already set to the right value? pods are getting deleted and recreated by daemonset-controller when the scaler is enabled (as was also seen in ). Why this is happening? One thing to note here - all those delete calls are preceded by calls from respective kubelets (but maybe that's expected). cc /priority critical-urgent /assign
Marked this as release-blocker as at least some part of it seems like a correctness issue (i.e fluentd pods dying and getting recreated). Note also that this is causing increased mem-usage with daemonset-controller (depicted here - )
Thanks for filing this! The only thing that was changing (apart from ds/fluentd-gcp status) was the resource version.
I modified fluentd scaler to use --dry-run in to check whether there would be a change, but this still keeps on updating, even though there is no change. Apparently, the logic inside kubectl is broken...
Btw, addon manager sends tons of PATCH requests for all resources every minute anyway, so even though I will fix fluent-scaler not to send them, it won't solve everything.
Should this issue actually be closed? It was auto-closed by the bot, but nobody has confirmed that the actual issue is fixed.
I'll let to confirm (looks like he'll be back on Monday)
/status in-progress
Yes, we should keep this issue open until we fully understand what's going wrong and fix the root-cause. I believe much of the discussion wrt this is happening under related issue
ACK. In progress ETA: 20/03/2018 Risks: May delay the release if not satisfactorily resolved
Need a number of successive green runs, then close.
So I took a look and seems like the fluentd-scaler version bump (which was supposed to fix) was merged b/w runs and of our . However, I don't see any visible improvement in controller-manager memory usage around that time: !
Also, I'm noticing such lines occasionally (once every 5-10 secs in the start and once every ~1min later on) from of latest run: And back when the fluentd-scaler wasn't doing anything useful (i.e before ), the had such lines only sparingly.
Now it isn't even sending requests (other than GET) to the apiserver. How do you generate that graph? Is it 99th percentile of memory usage over time? Or what exactly? One difference after introducing fluentd-gcp-scaler is that now there is a change occurring on the DaemonSet (once, but still). Also, what test is failing because of this? I'd like to take a look into the logs.
Answering your questions: I fetched the ResourceUsageSummaries for density test from historical runs (e.g ) and then obtained the 99% mem-usage for controller-manager for each run. Yes. More precisely, we're collecting mem-usage from controller-manager once every 1 min (or sth like that) and then taking 99th %ile of the resulting sample. Density and load tests. Where we're checking that mem usage of controller-manager is less than the given threshold (i.e 500MB). Ref: The tests are occasionally failing due to just crossing the threshold. Overall, the mem-usage had seen an increase back during the scaler change (see graph above). The job is this - https://k8s- Inside , you'll find all necessary logs (also a file named which has the resource-usage data).
The fluentd-scaler update eliminated fluentd restarts problem. Verified from controller-manager logs that we no longer see such logs that we were : This is great, and I believe that moves this out of release-blocker as we no longer face problems I mentioned in
That said, and me will continue looking into increase in mem-usage (which is not that serious). We'll start by verifying that turning the scaler off improves mem-usage.
Feel free to override me if you have a good reason :)
I agree that this doesn't sounds to be release-blocker.
But I would like to see follow-ups at least for issues like: does this memory increase come from? what was claimed to be "no-op" changes for daemon set were causing problems [is that issues in kubectl, apiserver, somewhere else?] Something like that may hit us again in the (near?) future if we won't understand it now. The current solution is really volatile [if there is a bug somewhere we shouldn't workaround it on our side, but instead have this fixed] at the cluster start (maybe it's fine to leave as is, but I would like to see a discussion of that and a conscious decision being made here)
Definitely agree those are questions we need to answer.
Yup. 1) is tested in . 2) I believe apiserver sends watch events even when only resource version changes, which causes the daemonset controller to recreate the pods. So either: resource version shouldn't change when there is no change anywhere else in the resource apiserver shouldn't send watch events when only resource version is changing daemonset should check whether watch event it got is actually changing anything 3) Isn't that an inherent part of the way addon manager works? It creates all sorts of resources, some dependent on the other (e.g. cluster roles and cluster role bindings), all at once. If some dependency is not satisfied, it will get fixed during the next run.
This is the code that is supposed to protect us from situation that something must change when we write object to etcd (resource version is not stored in etcd): So there should never be a situation that only resource version is changing. If it is, there is bug and need to be fixed. - FYI Reconciliation is a different thing that creating something in one state only to change that to a different state by a different component. I'm not saying this is for sure wrong approach. But I would like to see a conscious decision process discussing pros-and-cons of that.
That's what I'd expect as well. This I'm not sure if should be correct behaviour as IIUC watchers need to receive every RV update for correctness reasons ( ) Maybe. But are we sure if RV is just increasing without any object change? For creation of addons, yes (that race will likely last only for starting few seconds).
But this is wrt addon changing cluster resources, which can potentially create race at arbitrary time after cluster start (though your fix may have gotten rid of this).
Ok.. Seems like I missed Wojtek's comment above.
So I just checked the ds-controller code and it seems like we're updating the ds status only if this condition is being violated: Which means one of those fields is probably mismatching. I'll try to add some logs and run the test at --v=4 to get more info.
anything is possible, but we do have tests ensuring this works properly (no-op updates do not actually result in an update+resourceVersion bump in etcd):
So I ran some 100-node experiments against with more debug logs and I think I now understand the reason for this. The controller-manager logs and mainly show the following difference: This logs show-up when the scaler is active. And these pod deletions are happening as part of rolling-upgrade of daemonset to newer version (the one having resources set). You can see they're getting deleted one-by-one, which also explains why it takes so long (and so are happening even 30 mins after the cluster creation). And the reason why they're deleted one-by-one is because ds-controller has a rolling-upgrade strategy where by default. Proof: And the increased mem-usage FMU is because of the rolling-upgrade performed by ds-controller (over the prolonged period) - as it's the only difference I observe from logs Further, the reason why the RV was increasing was because of few fields 'NumUnavailable', 'NumAvailable', etc fluctuating while the rolling-upgrade
I spoke offline with and suggested that we should increase maxUnavailable for the fluentd daemonset to a large enough value so that we're not bottlenecked by it. My reasoning is: fluentd is not some user workload which will have some service downtime in case there are many unavailable at once fluentds on nodes are independent of each other (i.e one fluentd shouldn't need to wait for another node's fluentd to come up) we should try to make the scaling happen as fast as possible - otherwise it may be too late before they cope-up with increased log-load in the cluster I'm going to make that change and test it against my PR (thanks for pointing that we can change maxUnavailable directly in ds config).
So increasing maxUnavailable indeed seems to help (ref: ): All the 101 replicas were deleted at once instead of one-by-one and happened pretty much within 2 mins of the original fluentd-ds seen by the controller: We should get that PR into 1.10.
Thanks for confirming this! Did increasing maxUnavailable also reduce c-m memory usage?
So the run was still over the threshold, but I was expecting it as I ran c-m with increased verbosity (which ). That said, I'll observe how the usages are once that PR gets in - and FMU it's still possible to have higher values due to the rolling-upgrade potentially happening during the test.
Is there anything else to do here or should this be closed now?
Let's close it once we're sure there are no more such failures. We may also need to bump the threshold. On Mon, Mar 26, 2018, 9:46 AM Kubernetes Submit Queue < :
So bumping maxUnavailable for fluentd rolling-upgrade in is causing couple of issues: Our latest 5k-node performance failed as part of BeforeSuite where we're checking that ~all kube-system pods are running or ready. The reason is because quite some fluentd pods were killed as part of the rolling-upgrade and we timed out before all those pods could come up Our 100-node performance runs due to high CPU/mem-usage of c-m observed during density test, in cases when the fluentd upgrade happens during the test - which is kind of expected So at this point, we might want to choose among one of the 2 options: Rollback , which comes at the cost of tradeoffs I explained in Increase our CPU/mem thresholds to eliminate those flakes in 100-node job and increase the "max-allowed-not-ready-pods" in kube-system ns to a higher value, so that our 5k-node test doesn't fail in BeforeSuite Thoughts?
TBH, I don't know why it was merged without waiting for opinions... I really vote for reverting that. We don't want this - this may mean making cluster "not fully usable" from user perspective.Which is not really what we want to test.
SGTM. Let's revert it and increase our thresholds for the time-being. The discussion about changing fluentd rolling-upgrades can continue to happen in the background.
[MILESTONENOTIFIER] Milestone Issue Needs Approval Action required: This issue must have the label applied by a SIG maintainer. If the label is not applied within 5 days, the issue will be moved out of the v1.11 milestone. <details<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Escalate to the issue owners and SIG owner; move out of milestone after several unsuccessful escalation attempts. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
This issue is now fixed after my above PRs. We're having continuous for the job
In and , I started seeing these flakes. The job in question has not been very flaky before so this is odd. In and , I started seeing these flakes. The job "Job should not create pods when created in suspend state" is the one that is flaking. Triage says a few days ago. No response No response /sig apps
Did some digging and I did not see any changes that could account for an increase of flakiness in the job controller code. There was a change made to the e2e tests: The logs say: The code is pretty simple so I wonder if we are hitting a case where sometimes the suspend condition is not set and we fail. Maybe we could add a consistent call here?
/triage accepted /priority important-soon
/cc
/reopen /assign do you mind monitoring the CI state for 1 or 2 days to confirm your PR really fixed it ?
Reopened this issue. <detailsIn response to : /reopen Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
It makes sense to me.
cc I think this issue has been fixed. Can we close it?
/close Thanks for the fix. The test is not flaky now.
Closing this issue. <detailsIn response to : Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
We're seeing an issue with large memory growth in NoExecuteTaintManager when pods with tolerations are rapidly created/deleted. The item to the workqueue contains the full pod objects received from the informer: That means those pod objects are retained until they are processed. Additionally, it looks like the processing loop never removed the keys from the workqueue when it was done. It also means that multiple updates for the same pod result in multiple items in the workqueue, rather than enqueuing the pod to be processed once, like other controllers do. Simplifying this controller to enqueue pod updates only by name/namespace, and operate on the latest pod obtained from the store (like other controllers do) would speed up the processing and relieve the memory burden of the workqueue. /sig scheduling /cc
/assign
Thanks for finding this Jordan - looking into your fixes.
the memory issue was fixed by /close
<!-- This form is for bug reports and feature requests ONLY! If you're looking for help check and the . If the matter is security related, please disclose it privately via --Is this a BUG REPORT or FEATURE REQUEST?: What happened: fsgroup is only intended for RWO volumes because it recursively changes the permissions of all the directories in the volume. This can cause problems for nfs-type volumes because multiple pods could access the same volume and have the permissions changed out from underneath. What you expected to happen: Determine whether or not the plugin supports fsgroup either via: access mode. This is still a bit ambiguous for nfs volume types, that can theoretically support rwo, especially since Kubernetes does not do any sort of access mode enforcement. a new csi capability
CC
Are you saying that CSI internal driver should check driver capability for RWO, if supported, only then apply value ? What would the be ?
Yes one possible way is to infer fsgroup is supported for all rwo volumes, which may not be true? Like a nfs volume technically supports rwo but not fsgroup. So the other option is to make it a new CSI capability. I'm not sure exactly sure how best to describe it though, maybe something about volume ownership management.
Will it work if is not set then don't chown the ownership? I would think that for shared file system types(nfs, glusterfs) - won't be set. For raw block storage too - I think we shouldn't have to do this.
what do you mean by ? Shouldnt have to solve this issue ?
I meant - which is obvious. I just meant that the entire mechanism of changing ownership of files should apply only to volume types with block storage filesystems. I am just wondering if we can get away with detecting fsType and not have to introduce new capability in CSI.
agreed, do not map well to current csi.Capabilities enums which makes it hard to determine RWO when mounting. Perhaps, instead of a new capability, we revisit the existing capabilities to see if there are any way RWO can be reliably inferred from returned capabilities.
Currently, this is what the code is doing which we knew needed to be revisited:
Users may not fully understand what fsgroup is and set it anyway, or a default psp with some fsgroup may be defined in which it will automatically be set on all pods. The first case we can attribute to user error, but the second case may not have a good workaround and could cause permission issues on rwx volumes. So I think we do need to solve this.
Yes, agreed this has to be solved. As you pointed out user error or improperly configured PSP can set value that does not match the correct attribute, causing permission issues. Here is what I think can be done without new CSI capabilities: , then do not apply fsgroup then: , apply fsgroup , do not apply fsgroup to void stated issue then: , apply fsgroup I think that may help deduce when to apply fsgroup.
If mount reference count <= 1, apply fsgroup If mount reference count 1, do not apply fsgroup to void stated issue A RWM volume (say NFS) can be mounted on multiple nodes. Do you have reliable count of mounts across all nodes?
In addition, that would be a change in semantics to how we support rwx volumes today. If we don't have a new CSI capability, then I think the best course of action is to only apply fsgroup for RWO volumes. Some questions remain: Do all RWO volumes support fsgroup? RWX is a superset of RWO. So if a user requests a RWO volume, it could still be satisfied by a RWX volume type. Can we detect this and still not apply the fsgroup even though the volume capability will say RWO?
If there's not a reliable way to enumerate node mounts (thinking about it, for CSI external drivers, the answer is NO). Theres is no reliable way to have k8s autocorrect a bad fsgroup/PV.AccessMode combo. So we would have to rely on the config provided by user/admin. I have come to similar conclusion as -- mainly, apply fsgroup only to volumes with access modes. The other k8s specified access modes, , cannot be reliably applied.
Even if we were to introduce additional plugin capabilities that the CO can query (from driver) to find out what modes can be applied, there would still be the possibility of user/admin mis-configuration. Probably not, Is this where you think an additional capability would help ( ) That is a good point. Can't think of a way to detect/guess RWX when RWO is specified, this without the driver providing that info. Can we adopt the rule until we decide what capabilities would look like ?
I will just try to summarize the conversation from Slack. We can safely assume that only applies to block storage type with file system on top of it. So for all - volumes that need to have called MUST have a file system() on them. Is that assumption correct? If it is - can we not use presence of field or we can query the local volume for fs type (like how we query selinux relabelling capability) and call only when - volume is RW and has a valid block storage file system type. Will this not solve our use case? EDIT: renamed to .
We'll need to make it clear that is only for block-based storage systems, and the field cannot be reused for some multi-protocol file server (ie, nfsv3, nfsv4, smb, etc). For that case, then the plugin will need to expose its own new storageclass parameter + volume attribute to specify protocol.
Can we not use to get fsType - and use that information for calling ? It may be problematic to rely entirely on present inside csi source.
the only problem with is that in CSI volume info is intentionally opaque to the CSI code inside k8s. All mount logic is delegated to the driver.
Instead of a new capability or trying to deduce how best to apply fsgroup, why not pass the to the driver and let it decide how to apply it (either as a attribute for the mount or top level request param)? Right now, all mount logic has been delegated to the external CSI driver except for applying permission. Thoughts ?
that is true but we can still inspect file system on the mounted path because by the time gets called volume is already mounted. This should be no different than how selinux support is determined. what you think?
is this approved for 1.12 milestone? Thank you for working on this,
still discussing how best to approach this.
It would be a 1.12.
[MILESTONENOTIFIER] Milestone Issue: Up-to-date for process <details open<summaryIssue Labels</summary- : Issue will be escalated to these SIGs if needed. : Never automatically move issue out of a release milestone; continually escalate to contributor and SIG through all available channels. : Fixes a bug discovered during the current release. </details<details<summaryHelp</summary<ul<li<a href=""Additional instructions</a</li<li<a href=""Commands for setting labels</a</li</ul</details
Approved for 1.12
Based on previous post, this is a summary: If then is not applied (because it could be an indication of a nonblock fs, or error) Done is provided if fstype is invalid Done if fstype is valid and then apply fsgroup logic else ignore fsGroup Done
- any updates on ?
Waiting for a review and approval.
you have a outstanding comment on the PR. Can you please answer that?
How does this approach relate to inline CSI volumes? There's no PersistentVolume involved, and hence no way to specify an access mode. , but my driver ignores it (always using tmpfs). I considered having my CSI driver read the mounting pod's spec to see if it has a security context with "fsGroup" set, since that detail is not supplied to the driver, as . My goal is to have the volume mounted with the files on it readable by the container's user, but there's no way for my driver to know who that user will be. Am I supposed to just use 0444 for the ownership bits, to allow everyone to read the files?