Change:
### Version removal <del> An older API version cannot be dropped from a CustomResourceDefinition manifest until existing persisted data has been migrated to the newer API version for all clusters that served the older version of the custom resource, and the old version is removed from the `status.storedVersions` of the CustomResourceDefinition. </del> <ins> An older API version cannot be dropped from a CustomResourceDefinition manifest until existing stored data has been migrated to the newer API version for all clusters that served the older version of the custom resource, and the old version is removed from the `status.storedVersions` of the CustomResourceDefinition. </ins> ```yaml apiVersion: apiextensions.k8s.io/v1

Discussion:
This is a Bug Report Problem: One part of the "Versions in CustomResourceDefinitions" document says that conversion is performed from storage version to served version. Another says it's not (and furthermore, that k8s will lie about by setting it without actually making any attempt at schema conversion). Specifically: says: the same time says: The above sentence is not true in general. Specifically because the part: blurs the distinction between the "persisted object" (which indeed never changes on reads) and the served representation (which definitely is converted, if only to change the apiVersion string), this sentence could be read in two ways: persisted object is not changed on disk. The served representation is not converted (i.e. does not differ from the persisted object) other than the version string. This first interpretation is only correct if the schemas do not differ. A casual reader of a general document on conversion webhooks certainly dos not have this assumption, and definitely not before reading the example which follows the above sentence. persisted object is not changed on disk. The only conversion applied to persisted object is that the apiVersion string is changed. This conversion to persisted object happens when the request is being served. This second interpretation is just plain wrong. The on-disk object does not change. Period. Then later, there is an example which describes a special case of serving a representation of an object persisted at version: This sentence is correct, and in this particular context the first sentence makes more sense (but is still blurry). Proposed Solution: the first sentence to: match what happens in the general case of stored version != served version. Not just in the specific -case. make an explicit distinction between (conceptually immutable on reads) persisted object and the served representation, rather to try and describe both in one go an example for the case where conversion webhooks are actually useful (i.e. where the schema between persisted and served versions differs). This can be an additional second example, or the existing example can be replaced. [separate issue] provide an additional example of what happens when there is no conversion webhook defined and schemas differ schemas are identical (except for the version string) [separate issue] additionally, it might be good to use "stored version" consistently, rather than "persisted version". This should also probably be a different orthogonal issue.

Page to Update:
The difference is that the persisted object is not changed no matter what version of its API you use to read it. That's what the second quote is about. If you have a persisted object and try to use a different version of that API to read or modify it, then the API server must convert its representation the persisted object to the API version that the client asked for. That's might involve a webhook for a CRD, as the first quote explains. GitHub (this website) does something similar. This issue is in no matter which API you use to access it. If GitHub adds a new kind of API for accessing issues, they might choose not to replace all their stored data; instead, they keep storing the data they have and convert on-the-fly when you access it. I think the pages are accurate. It's a difficult concept to explain well and if you have suggestions for improving the explanation (without changing the technical meaning), we'd be keen to hear them. /remove-kind bug /sig api-machinery /language en PS Using “persisted” vs “stored” consistently could be a useful cleanup - perhaps for a separate issue.
BTW, the reason that the two objects in differ only by the API version is because that was the only change you made. In Kubernetes, the convention when promoting an API to stable (taking or out of the version) is that you try not to change that API when you do so).
thanks for the fast response! Thank you for clarification, what you now described matches my understanding. I think the confusing part was this: I'm guessing this is indeed the case when is . But since the preceding sections talk at length about using conversion webhooks, such statement at this point was a bit of a shock to me. :-) I re-read the second quote and now that I have your clarification I admit it makes sense, if one assumes that the schema for the two versions is the same, as you say: Indeed the example that follows ("To illustrate this ...", changing to ) provides a hint to that, to someone who knows the semantics of version strings in Kubernetes. However this is a bit implied, and I think it would be better to make it explicit - as you say - this is a difficult concept.

I think there are two ways to improve the situation: the section mostly as is, but explicitly mention that it is talking about promoting to a version without changing the schema, where conversion webhook would indeed be unnecessary. Perhaps also linking to a document which explains the versioning conventions, i.e. that changing from to should imply no schema changes, unlike -. more general, and explicitly describe what happens in the following cases: schemas differ, and there is a conversion webhook (response is converted, but stored object stays intact) schemas differ, but there is no conversion webhook (that was actually what I was looking for in this document, since this is a corner case with potential for data loss in some cases) no schema difference and no conversion webhook (just is bumped in the response, but stored object stays intact) I think the latter option would be much more informative, however I lack knowledge about what happens in the middle case. If someone could chime in with a link to the relevant place in the code, I'd be more than happy to send a doc improvement PR!
Issues go stale after 90d of inactivity. Mark the issue as fresh with . Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with . Send feedback to sig-contributor-experience at . /lifecycle stale
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
also /wg api-expression
/remove-lifecycle rotten
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
in light of the discussion so far, would you be willing to revise the issue description to outline the defect you see or the improvement you'd like made?
finally had some time to reword the issue description. Please take a look and if this makes sense I can propose a patch and file the additional issues.
Thanks /triage accepted /priority backlog
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
/remove-lifecycle stale
When working on a PR to address this issue, I tried to find some documentation in to back the following claim: but did not find anything concrete. Do you know if this goal is actually written down somewhere, or is this tribal knowledge?

That's not a policy, it's an observation. SIG Architecture might have an opinion to add here that it's also an aim of the Kubernetes project. /sig architecture