Change:
We need to define the CRI API versioning policies: we bump the version of API when methods and optional fields are ? CRI API need to indicate that the new methods are optional (not part of a conformance) for a new features that are in alpha/beta? the release cycle for the features going thru alpha/beta stages that require CRI API changes and implementation in various runtimes. CRI API allow "optional features like suggested here: Would this needs to be tested as part of k8s e2e to make sure k8s doesn't depend on "optional" APIs. Also we need to define the deprecation and feature removal policies. We need to define clear rules of working with CRI API. /sig node /cc /priority important-longterm
+1 Thanks for raising this issue.
Great point, I'm in favor of this. With v1 we're in shape to define the right set of policies. To clarify, when speaking about the version, we're referring to the runtime version defined there: No, because in protobuf 3 all fields are optional and we should only bump the CRI version on breaking changes. Adding new services (methods) which are not supported by the runtime would return an "Unimplemented" error, which can be handed by the kubelet. Yeah, by adding documentation to k/website?
IMO no. While I understand the "we linked them together and it didn't work" problem, that feels like a problem of their own making. The deps constraints (implicit or explicit, I don't know) are being violated. Now, we COULD CHOOSE to jump through hoops to make this better - is it worth the energy to do this? Once we do it, it will be hard to stop doing it. E.g. defining a new for every method we add. Hopefully we don't add a lot of methods... I don't understand what you mean here? Yes. This is the biggest issue with out-of-tree implementations - adding a feature to the API is easy, but getting impls to support it, then cut a release, etc is harder. We lack a clear set of rules - do they HAVE TO support a feature? Do they have to support it pre-GA? Will we block GA contingent on impl support? How do we get feedback on new features? This problem exists in networking and storage, too. Unless we plan to require every impl to support every feature, we need some way to present and/or query what works and what doesn't, right? Otherwise we end up with impls that are lacking feature X but still accept the API for it - a recipe for unhappy users.
Adding in a reference to https://gateway-
For storage we make most features optional. If in the future we require some new feature (this hasn't happened yet), I think we would have to bump the major version. User discoverability of what optional features an implementation supports remains a challenge. Today each implementation is responsible for documenting which features they support. We have a to list implementations, but it's up to each implementation to keep their entry updated. Right now if a user tries to use an optional feature that the implementation doesn't support, they will get feedback ranging from a nice error message to no progress and no response.
I think it might be interesting to have at least a common pattern for this.
I have started this document for the discussion:
/triage accepted
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Mark this issue or PR as rotten with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle stale [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues and PRs according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Mark this issue or PR as fresh with Close this issue or PR with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /lifecycle rotten [1]:
The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. This bot triages issues according to the following rules: After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closed You can: Reopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Please send feedback to sig-contributor-experience at . /close not-planned [1]:
Closing this issue, marking it as "Not Planned". <detailsIn response to : The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. After 90d of inactivity, is applied After 30d of inactivity since was applied, is applied After 30d of inactivity since was applied, the issue is closedReopen this issue with Mark this issue as fresh with Offer to help out with [Issue Triage][1] Instructions for interacting with me using PR comments are available . If you have questions or suggestions related to my behavior, please file an issue against the repository. </details
/remove-lifecycle rotten /reopen
Reopened this issue. <detailsIn response to : /remove-lifecycle rotten /reopen </details

Discussion:
Read more about CRI API at [kubernetes docs](https://kubernetes.io/docs/concepts/architecture/cri/). The repository [kubernetes/cri-api](https://github.com/kubernetes/cri-api) is a mirror of https://github.com/kubernetes/kubernetes/tree/master/staging/src/k8s.io/cri-api. <del> Please do NOT file issues or submit PRs against the [kubernetes/cri-api](https://github.com/kubernetes/cri-api) </del> <ins> Please do **not** file issues or submit PRs against the [kubernetes/cri-api](https://github.com/kubernetes/cri-api) </ins> repository as it is readonly, all development is done in [kubernetes/kubernetes](https://github.com/kubernetes/kubernetes). <ins> The CRI API is defined in [kubernetes/kubernetes](https://github.com/kubernetes/kubernetes) repository and is **only** intended to be used for kubelet to container runtime interactions, or for node-level troubleshooting using a tool such as `crictl`. It is **not** a common purpose container runtime API for general use, and is intended to be Kubernetes-centric. We try to avoid it, but there may be logic within a container runtime that optimizes for the order or specific parameters of call(s) that the kubelet makes. ## Version skew policy On a single Node there may be installed multiple components implementing different versions of CRI API. For example, on a single node there might be: - _Kubelet_ may call into _Container Runtime_ (e.g. [containerd](https://containerd.io)) and _Image Service Proxy_ (e.g. [stargz-snapshotter](https://github.com/containerd/stargz-snapshotter)). _Container Runtime_ may be versioned with the OS Image, _Kubelet_ is installed by system administrator and _Image Service proxy_ is versioned by the third party vendor. - _Image Service Proxy_ calls into _Container Runtime_. - _CRI tools_ (e.g. [crictl](https://kubernetes.io/docs/tasks/debug/debug-cluster/crictl/)) may be installed by end user to troubleshoot, same as a third party daemonsets. All of them are used to call into the _Container Runtime_ to collect container information. So on a single node it may happen that _Container Runtime_ is serving a newer version'd kubelet and older versioned crictl. This is a supported scenario within the version skew policy. ### Version Skew Policy for CRI API CRI API has two versions: - Major semantic version (known versions are `v1alpha2` ([removed in 1.26](https://kubernetes.io/blog/2022/12/09/kubernetes-v1-26-release/#cri-v1alpha2-removed)), `v1`). - Kubernetes version (for example: `@1.23`). Note, the `cri-api` Golang library is versioned as `0.23` as it doesn't guarantee Go types backward compatibility. Major semantic version (e.g. `v1`) is used to introduce breaking changes and major new features that are incompatible with the current API. Kubernetes version is used to indicate a specific feature set implemented on top of the major semantic version. All changes made without the change of a major semantic version API must be backward and forward compatible. - _Kubelet_ must work with the older _Container Runtime_ if it implements the same semantic version of CRI API (e.g. `v1`) of up to three Kubernetes minor versions back. New features implemented in CRI API must be gracefully degraded. For example, _Kubelet_ of version 1.26 must work with _Container Runtime_ implementing `k8s.io/cri-api@v0.23.0`+. - _Kubelet_ must work with _Container Runtime_ if it implements the same semantic version of CRI API (e.g. `v1`) of up to three minor versions up. New features implemented in CRI API must not change behavior of old method calls and response values. For example, _Kubelet_ of version 1.22 must work with _Container Runtime_ implementing `k8s.io/cri-api@v0.25.5`. ## Versioning This library contains go classes generated from the CRI API protocol buffers and gRPC API. The library versioned as `0.XX` as Kubernetes doesn't provide any guarantees on backward compatibility of Go wrappers between versions. However CRI API itself (protocol buffers and gRPC API) is marked as stable `v1` version and it is backward compatible between versions. Versions like `v0.<minor>.<patch>` (e.g. `v0.25.5`) are considered stable. It is discouraged to introduce CRI API changes in patch releases and recommended to use versions like `v0.<minor>.0`. All alpha and beta versions (e.g. `k8s.io/cri-api@v0.26.0-beta.0`) should be backward and forward compatible. ## Feature development Some features development requires changes in CRI API and corresponding changes in _Container Runtime_. Coordinating between Kubernetes branches and release versions and _Container Runtime_ versions is not always trivial. The recommended feature development flow is following: - Review proposed CRI API changes during the KEP review stage. Some field names and types may not be spelled out exactly at this stage. - Locally implement a prototype that implement changes in both - Kubernetes and Container Runtime. - Submit a Pull Request for Kubernetes implementing CRI API changes alongside the feature code. Feature must be developed to degrade gracefully when used with older Container Runtime according to the Version Skew policy. - Once PR is merged, wait for the next Kubernetes release tag being produced. Find the corresponding CRI API tag (e.g. `k8s.io/cri-api@v0.26.0-beta.0`). - This tag can be used to implement the feature in Container Runtime. It is recommended to switch to the stable tag like (`k8s.io/cri-api@v0.26.0`) once available. ## Change history Here is the change history of the Container Runtime Interface protocol: ### v1.20 `git diff v1.19.0 v1.20.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - CRI [v1 introduced](https://github.com/kubernetes/kubernetes/pull/96387) ### v1.21 `git diff v1.20.0 v1.21.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` No changes ### v1.22 `git diff v1.21.0 v1.22.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - [Windows host process support](https://github.com/kubernetes/kubernetes/pull/99576) - `PodSandboxConfig` has `windows` field  of type `WindowsPodSandboxConfig` - New type `WindowsPodSandboxConfig` introduced - New type `WindowsSandboxSecurityContext` introduced - The type `WindowsContainerSecurityContext` has a new `host_process` boolean field - [Feature: add unified on CRI to support cgroup v2](https://github.com/kubernetes/kubernetes/pull/102578) - The type `LinuxContainerResources` has a new field `unified` which is a map of strings - [Alpha node swap support](https://github.com/kubernetes/kubernetes/pull/102823) - The type `LinuxContainerResources` has a new `memory_swap_limit_in_bytes` int64 field ### v1.23 `git diff v1.22.0 v1.23.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - [CRI: add fields for pod level stats to satisfy the /stats/summary API](https://github.com/kubernetes/kubernetes/pull/102789) - New functions `PodSandboxStats`, `ListPodSandboxStats` with the corresponding types of request and response objects are introduced - [pass sandbox resource requirements over CRI](https://github.com/kubernetes/kubernetes/pull/104886) - New fields on `LinuxPodSandboxConfig`: `overhead` and `resources` of type `LinuxContainerResources`. - [prevents garbage collection from removing pinned images](https://github.com/kubernetes/kubernetes/pull/103299) - The type `Image` has a new boolean field `pinned` ### v1.24 `git diff v1.23.0 v1.24.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - [Update CRI-API Capabilities to include a field that allows us to set ambient capabilities](https://github.com/kubernetes/kubernetes/pull/104620) - The type `Capability` has a new string field `add_ambient_capabilities` - [CRI-API - Add rootfs size to WindowsContainerResources](https://github.com/kubernetes/kubernetes/pull/108894) - The type `WindowsContainerResources` has a new int64 field `rootfs_size_in_bytes` ### v1.25 `git diff v1.24.0 v1.25.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - [kubelet: add CRI definitions for user namespaces](https://github.com/kubernetes/kubernetes/pull/110535) - The new type `UserNamespace` introduced to represent user namespaces id mapping - The type `NamespaceOption` has a new field `userns_options` of type `UserNamespace` - [Minimal checkpointing support](https://github.com/kubernetes/kubernetes/pull/104907) - The new method `CheckpointContainer` introduced with the corresponding request and response types - [Update CRI API to support Evented PLEG](https://github.com/kubernetes/kubernetes/pull/111642) - The new streaming method `GetContainerEvents` is introduced with the corresponding request and response types - [CRI changes to support in-place pod resize](https://github.com/kubernetes/kubernetes/pull/111645) - The new type `ContainerResources` is introduced - The type `ContainerStatus` has a new field `resources` of type `ContainerResources` - The semantic of `UpdateContainerResources` updated. The method must be implemented as synchronous and return error on failure ### v1.26 `git diff v1.25.0 v1.26.0 -- staging/src/k8s.io/cri-api/pkg/apis/runtime/v1/api.proto` - [CRI: Add Windows Podsandbox Stats](https://github.com/kubernetes/kubernetes/pull/110754) - Added fields to the type `WindowsPodSandboxStats` expressing stats required to be collected from windows pods. - [Windows hostnetwork alpha](https://github.com/kubernetes/kubernetes/pull/112961) - New type `WindowsNamespaceOption` introduced - The type `WindowsSandboxSecurityContext` has a new field `namespace_options` of type `WindowsNamespaceOption` - [Improve the API description of `PodSecurityContext.SupplementalGroups` to clarify its unfamiliar behavior](https://github.com/kubernetes/kubernetes/pull/113047) - Clarified the expected behavior of `SupplementalGroups` field of `PodSecurityContext` - [Add Support for Evented PLEG](https://github.com/kubernetes/kubernetes/pull/111384) - The type `ContainerEventResponse` updated: the field `pod_sandbox_metadata` removed and fields `pod_sandbox_status` and `containers_statuses` added. - The type `PodSandboxStatusResponse` has a new fields `containers_statuses` and `timestamp` </ins> ## Community, discussion, contribution, and support Learn how to engage with the Kubernetes community on the [community