jiebi/CodeConvo huggingface dataset:
After my repro and investigation(really spent a lot of time...), I found the issue lies here, would return empty even after granting sp access to azure acr, and after around 15min, it returns correct loginServer list: Looks like it's a SDK or acr list with specific sp issue, workaround is wait around 15min or more time, it could recover by itself. Paste my investigation logs here, I grant sp access to acr in around , you could see it returns empty registry list first and then after a few more minutes, it returns the correct list with loginServer:
BTW, in v1.12, acr list() uses new API, I will check whether in v1.12 would have same issue. Acutally I have already implemented LazyProvide func(check here: ), while it won't fix the original issue.
Update: After a few more tries, I found it's related to this code snippet, before , we need to generate a new ServicePrincipalToken every time: I will move this code snippet into Provide() func, and send out a PR! The failure recover time is within 2min now.
Seccomp profile is passed to runtime by in CRI: is relative to node's local seccomp profile root, it is defined in kubelet. But runtime doesn't know it. We should pass full profile path in CRI. cc/ Will file a PR to fix this later.
Add support for Seccomp sounds like a new feature. The milestone should be reset to , correct?
not really. The current implementation supports seccomp already. If it's missing critical information to function properly, this would be a bug.
are we blocking releases for CRI in 1.5? i was not sure, so marking this as non-release-blocker and leaving 1.5 milestone. please correct if i am mistaken.
I don't get why this is a 1.5 targetted thing myself. Yes we want CRI to be in a good state in 1.5, but afaik we have zero intention of anyone actually using (at the very least because of the lack of exec auth regressing kubelet security). why should this be part of 1.5?
First, this is a small enough change that allows non-docker integrations to properly support seccomp. Second, if "we have zero intention of anyone actually using" is a valid reason, we should never fix any bugs in CRI. We do want people to try using it (perhaps not in production yet). However, given that seccomp is only in alpha, I am ok with dropping this and just continue the docker support for now.
Perhaps I misunderstood the meaning of code freeze, but I thought it was only bug-fixes regardless of size of change (or exceptions that have gone through some process) What I meant is we don't intend this to be used generally as part of the 1.5 release. Because of that, I don't think we should consider bugs in CRI bugs for the purpose of the 1.5 release freeze. My statement is just that we shouldn't violate code-freeze for this. We should totally fix this right after the 1.5 freeze window ends, and saying that I'm arguing we should never fix anything is disingenuous and taking it to an absurd extreme.
the only thing we don't seem to agree on is whether seccomp support was considered to be supported for CRI in 1.5.
My interpretation is that we assumed seccomp would be supported until we found out that it wouldn't for other runtimes due to the lack of profile path. That's the only reason why I considered this a bug. Of course we could say that this is not really an important feature in CRI, so we can drop it, but we can be more clear on that.
If there's a bug in a feature no one uses, does fixing it make a sound? I think what we're not agreeing on is the role of CRI in 1.5. I understood it as something that is not generally usable. We can't recommend a customer use it, even as an experimental feature, due to security aspects (if nothing else). Correct me if I'm wrong. It's a developer integration point, and developers integrating with it are doing so with the kubernetes master branch, not with some release. Those reasons together result in my opinion that CRI is still not really a customer/release feature, and thus bugs or features related to it should not be exceptions to the freeze window, but rather should targeted at master as soon as the freeze lifts. If this were already post-freeze, would we be cherry-picking this over to 1.5?
This is not true. cri-o is depending on this fix for apparmor support.
is it depending on the 1.5 release branch directly, or on master?
I think it's better to make this in 1.5, so cri-o could work with first release with experimental CRI. It helps to encourage people to use CRI and find features misses or bugs.
Modifying the user-specified annotation is a little bit unsettling. I think adding a new annotation explicitly for the path would be better. Or, we should just add an explicit field for this. I spoke to and separately. I still think this is a bug fix, but we may want to be more careful about the fix. Given the time constraint, let's just list this as a known issue and work on it for 1.6.
OK, let's move this to next release and think carefully about seccomp feature besides this: kubelet may just pass the profile as string via CRI; may assume the profile locate in a file in a known format and pass path to the runtime; or maybe some other solutions.
We need to clearly define it. define profile format clearly
Please document known issues for 1.5 in
(copied from ) And let's move here to discuss how to pass profiles to runtime. Related with .
This is issue is related with two level APIs: CRI, since we may also want to support profiles from ConfigMaps, I think we should pass full profile contents instead of path to runtime. issues is also related with seccomp api in kube-apiserver, it is based on annotations today. We should define seccomp profile format clearly (see ) and decide whether to add struct types to pod spec or just continue the annotations way. I think it's more clear to add specific struct types to pod spec.
Agreed - the path stuff is really a crutch (and implementation-specific) - I think is the answer here.
I don't disagree, but it's not strictly a requirement for supporting ConfigMaps (kubelet could create a file with the contents). An open question is whether to pass it as a raw string, or wait for and pass the structured data. The annotations way is deprecated (). I think the bigger question is where/how to put the data. E.g. structured data in the API, unstructured (string type) data in the API, reference to another object (e.g. ConfigMap). If the Kubelet needs to parse the spec, I think that's a good argument for a structured data type (either a standalone object or embedded in the pod API).
Yes, already opened to replace annotations with typed structs. I personally prefer a structured data type in both kubernetes API and CRI. WDYT?
A counter argument is that as the spec grows or new features are , we need to keep updating the Kubernetes API to maintain parity, or update the code in each CRI implementation to translate the Kubernetes spec. OTOH, if we use unstructured data we can use 3rd party libraries to parse out the data we need (assuming the spec evolves in a backwards compatible way).
What spec?
Whichever seccomp profile spec is being implemented by runtimes. E.g. if we decide to base the Kubernetes seccomp profile API off the OCI seccomp profile spec, and OCI adds some new features, at some point we'd probably want to add those features to Kubernetes.
It also means that if different runtimes support different feature sets, we'll be stuck either tracking only the common features, or specifying features that may be ignored by the runtime. This may be a non-issue if the seccomp feature set rarely changes.
Well, yes, but that applies to everything in the CRI, and is one of the tradeoffs we ma[d|k]e by introducing this abstraction in the first place.
My $0.02: Moving the existing logic/information for seccomp into a first-class field for CRI (as in ) is productive because of the known problems with the annotation approach. My initial proposal for seccomp support included a resource similar to what has proposed in kubernetes/community and we were unable to come to an agreement on the issue refers to here: I do not necessarily expect that we will be able to come to an agreement about the seccomp resource in time for 1.7, so I think we should proceed with and do backward compatible CRI changes later if we need to inject the content of the seccomp profile instead of the path. I think we would deeply regret embedding API surface to describe seccomp profiles in the Pod API :)
disabled them since was flaky. We need to find and fix the issue and then reenable these tests. cc Related issues: and
Probably should be higher priority so it doesn't get dropped
The issue is that we dont wait for the resource to be deleted unless we are using a reaper: This breaks for federation since we use finalizers. kubectl should wait for the resource to be deleted. We should also set OrphanDependents = false. This will have no effect in kubernetes but will enable cascading deletion in federation. Ref cc Will send a fix.
Sent
Updated to remove the wait (GET) at the end so it no longer fixes this issue. Sent to update the test to include a wait which fixes this issue.
go1.18 dropped support for certificates signed with the SHA-1 hash function - While this does improve security, and publicly trusted Certificate Authorities no longer issue such certificates, it is possible that user-provided certificates issued by private Certificate Authorities and used in webhooks or aggregated API servers do still use this. Similar to removal of support for certificates without subjectAltNames in go1.17, we need to: [x] add metrics when webhook/conversion/aggregated certificates with this hash are encountered - [x] document use of and the metric from 1 as a mitigation for clusters which still need to identify and replace use of such certificates - to release note and known issues [x] add audit annotation when SHA-1 cert is encountered - [x] update docs at and - /sig api-machinery instrumentation docs /priority important-soon /milestone v1.24 /kind bug documentation Upgraded 1.23 clusters to continue working as-is Configure an admission webhook, conversion webhook, or aggregated server with a serving certificate signed with the SHA-1 hash function. Upgrade to HEAD of 1.24. The certificate will no longer work by default No response 24 HEAD (go1.18-based) n/a n/a No response No response No response
it looks like NewMissingSANRoundTripperWrapperConstructor already has all the plumbing, and instead of missing SANs we could check the SignatureAlgorithm and output a distinct metric
had a not-yet-tested WIP branch at with a sketch of what it might look like
/assign who offered to pick this up
GitHub didn't allow me to assign the following users: stlaz. Note that only , repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see <detailsIn response to : /assign who offered to pick this up Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
Indeed, please, you should trust Jordan :slightlysmilingface:
/assign
/triage accepted
These restrictions are on the GitHub side, this message is posted when it receives a specific error from the GitHub API while attempting to assign It is an annoying GitHub limitation, and one that seemingly does not exist in the web client vs the API.
adds the metrics. the release note on that issue describes the go change, links to go docs, describes use of the option, the new metrics, and the kube-apiserver logs what would you expect kubeadm to do with this? GODEBUG off by default and let users override GODEBUG on their own (is this possible)? GODEBUG by default on all clusters? GODEBUG by default only on upgrade? The risk of 1 is that a user upgrades without reading release notes and breaks a cluster that has SHA-1 certs in webhooks, etc The risk of 2 is that we propagate accepting insecure certificates longer, even in new clusters that we know don't have any pre-existing bad certs The risk of 3 is that it makes an upgraded 1.23 → 1.24 cluster and a new 1.24 cluster inconsistent
the majority of kubeadm users use kubeadm managed CA and certificates. there are less users that use external CA to sign their own certificates. sha1 attacks surfaced more than a before kubeadm's creation and Git hash collusion made the problem more known around 2017 - that was at a time when kubeadm did not support external CA. given these facts, an estimate on my side is that there are currently no kubeadm users that use sha1 certificates, but i would not exclude the possibility entirely. if such users exists, they are on they own and my vote goes for option 1, until they migrate their PKI to something secure. if the same users log tickets for us, our response would be "please use GODEBUG and migrate ASAP". all users should be able to see the notes in IMO, the PR needs an "action required" to appear on top of the release notes.
that's helpful, thanks agreed. weird, I set , but the label didn't get . the label manually.
and there's a way for them to set that envvar in kube-apiserver manifests when using kubeadm?
There's been a lot of intermittent github webhook delivery issues in the past few weeks or so, slash commands are going to be somewhat unreliable :/
they could use kubeadm patches to override the YAML manifest defaults - i.e. the contents of v1.Container -Env:
If I stuff into a file and use that after -f, it works fine. cc
What's on stdin?
Bug, I'm only accepting json because I naively reused watch partitioner and we don't have test cases. Will fix tonight.
If that's the problem, why did it work when I gave it that file not over stdin?
Different code path. Stdin supports streaming json a la watch (multiple json objects in a stream), so you can do Unfortunately I missed yaml input and didn't use a segmenting yaml decoder (returning a stream of yaml objects).
Ah! Thanks for explanation. I think it would be good for us to ask for e2e runs before merging kubectl changes, since there's not quite enough coverage in test- yet.
Yeah, I'm slacking on that for these changes.
Fix incoming.
Fix provided. More tests to test-cmd... :)
and server-side dry-run are being promoted to stable features in 1.18. Conformance should ensure that cloud providers support these features. As a user, I expect to work, which requires that cloud providers for cloud default webhooks. Conformance would generally avoid cloud-specific issues like this one: This suggestion is from /area conformance
/sig testing /sig architecture
/wg apply
There are some good references for conformance tests in the conformance meeting notes:
once merges, we need to promote the e2e test to conformance in a follow up PR, in accordance to this doc: It seems like we meet the conformance test requirements. Here's the process specifically, which includes sig-arch and a review board group:
From the recent runs of different scalability tests, I have a hypothesis that we are not reusing connections after "429" error is returned to the client. Here is description of one tests that I've seen failed: in 2000-node kubemark all watches were dropped at some point (that's a separate issue that is a bit unrelated here). happened next is that basically all watchers decided to do list + watch, and obvious a bunch of lists failed with 429 error (this is still expected). And we started retrying them. And this is still ok. that point, apiserver started consuming ~30 cores (which also isn't completely unexpected). what IS unexpected to me, is that looking into profile ~70% of the CPU (which was ~20 cores) were spend in "doFullHandshake". And this is a bit strange to me. Do we do something specific on 429 errors (e.g. close the connection) so that we won't be able to reuse it in the future? Or do you have any other hypothesis, why we were mostly doing "full handshakes" at that time?
On the server side, we can use on http server to get more info. Can we do the same thing on the client? I don't see anything obvious, but knowing when a connection from a client is closed (vs. terminated by the server, vs gracefully terminated via Connection: Close) might be useful in debugging these cases. The only thing I can think of is that the kubelet has multiple connections to the master, and so instead of reusing its pooled connections it's asking for new connections all at once (and blowing out the pool size in the client). Can we verify that the kubelet TCP connection pool size is greater than the default (which is 2?). Kubelet should definitely have a number of idle connections proportional to number of watches.
There's definitely a branch in the rest client where I think we don't fully read the response body on a 429. Though not explicitly stated in the docs, I think failing to read the entire body of a request also causes the TCP connection to not be reused. This is a code snippet from the HTTP client's redirect code: I can certainly add something like this to . I'm not sure if this is a problem for HTTP2 though.
was this running with HTTP2 enabled? I can't remember the status of that.
I'm pretty sure HTTP2 is off still. , krousey wrote:
Found the doc: I will go ahead and patch this.
Did we add a new error case that didn't catch this? Since this is the 3rd time we've fixed an edge case on reading the entire body, we should probably do what we can to make sure others don't show up.
It's the case where it's 429 or = 500 and there's a Retry-After header. I'm going to add it to the same place where we ensure that the body is closed so that it should always catch it.
k, thanks
Thanks for debugging it (and yes, http2 was switched off in my tests). And looking forward to see the fix. Thanks!
Background: kubectl and the Kubernetes reference docs both depend on the spec to operate on Kubernetes resources. The lacks important information required by kubectl and the docs. Usecase: kubectl can identify the patchStrategy and mergeKey for resource fields kubectl can identify the columns to print for on resources docs can parse and display the default value for fields docs can identify top-level objects Solution: Add items to the comment tag: Example: The long term plan is to remove the and struct tags once all code has been updated to use the swagger spec instead. Until then, comments should be for the struct tags and . Some verification process will need to be written to make sure the struct tags and comments remain in sync.
I think this wouldn't fall under sig-apps. cc
I would not do the columns stuff - we're moving get to the server side. Top level objects is already available in some form by the client - specifically the list that we use (should use) to get the swagger doc tells us what the top level type is (or is that for something like explain). Merge key, patch strategy, default values all belong as "real" attributes, although we had not necessarily intended those to be arguments to k8s:openapi-gen, but actual tags in their own right (like "+optional")
Thanks Is there a design proposal / eta for this? This is for generating the docs from swagger. Right now the list of top level resources for the docs to display is hardcoded in a . I would like to move the majority of the metadata out of the config file so that: same tool can generate reference docs for federated apiservers without writing a custom config. don't need to manually maintain and update the config for each release - e.g. adding new types I have no strong opinion about this as long as they appear in the swagger so it is possible right tools the can use this data. Can you comment on why openapi-gen vs specific tags?
Describe in the server makes sense to me. IIUC, just prints out a set of columns. Was the idea that the server would just print out the columns or return a list of column / values or something more complicated? No need to answer if it is in the issue / proposal already.
is the bigger issue that touches on a rough overview, I can't find my rambling brain dump of the problem, but get would be rough tabular json, you could also get that as csv, and describe could be a richer but not quite full structured key/value structure.
regarding tags, they do not need to be in the same k8s:openapi-gen tag, it can be repeated: they can also be separate tags but still general like this: The idea is to have a general way to add arbitrary extension to a type so we don't need to change code generator for each specific tag. I have no strong feeling about the format, but I would like it to be general vs tags like +optional. The difference is +optional is a defined tag in OpenAPI () but extensions does not have any meaning for code generator. They better to be general so generate can pass them without change.
No, I mean we should define conventions in our API code that match the intent (optional) rather than have tons and tons of repeated code that is setting x-kubernete-describe-columns (or similar)
I spent some time thinking about your comments last night. I don't think this is mutually exclusive with , and could be complementary. IMHO this is general metadata associated with the type, and including it in the doesn't preclude supporting an api endpoint that pre-parses and formats the data. However, including it in the swagger does allow clients using the swagger to take advantage of it. Following is an example of where the swagger would be useful even if get was in the sever. A generated Java client could use the display columns for the "toString" method of the resource. Another use case that had mentioned to me was to allow some kubectl commands to operate on directly on local files. e.g. (strawman) could print the columns. I don't think we want to do this now, but it is an example of why keeping this object metadata private to the server limits what we can do. My hope is that this swagger tags solution will also oprove portable to easily work with ThirdPartyResources as well by writing the display-columns as an annotation on the TPR and keeping the same format. is a can of worms, and I would agree that the solution is capable of providing a richer experience for aggregating information across multiple resource types - e.g.
how describing Nodes fetches Pod information. I still think that the solution is compatible with the richer solution you proposed and there is not much additional cost in doing it. Howv What would you think of supporting a set of optional ALPHA extensions for "display-columns" and "describe-columns" allowing any consumers of the swagger to print the objects. In theory it could also be used by the UI to provide a summary view without having to do separate lookups. As an ALPHA feature we could try it for supporting sig-service-catalog and revisit it in 1.7. If we consider it successful, the solution outlined in could become another consumer of the display-columns tag? I like this approach. My biggest concern here is being able to get this into 1.6. Since kubectl supports +/-1 version skew, we need will need to maintain the old logic for at least 1 full release after we add support for the new swagger based logic. That puts us at 6 months from now to fully migrate if we don't have any swagger extensions in 1.6.
I think that's a nice to have in the long run - but then people will say "why are the columns different when the server prints them from the local print"? And most of the pod columns in no way can represent the full complexity of what they print with swagger, so the best effort local printing is going to be really, really, really best effort. Also, local print is a really specialized case - if we just did the generic metadata print that would probably be ok. So while I agree it's valid, it seems like a lot of work for very little practical value. Again, not opposed, just don't want to do two things halfway and then have nothing. So just biased towards not adding more client complexity in the near term. So you want to hack this in for service catalog in 1.6 on TPR? I feel like that's a lot of hacks - the generic printer stuff at least gives you name and namespace, and I'd rather do nothing in 1.7 for generic client printer rather than jump ahead. I won't stop you if you want to spend time and effort on it, but it's a lot of specialized work that will then be neglected while we do the server side stuff.