Project: Firecracker (Runtime / Container Runtime)
File: api-change-runbook.md

This runbook will cover triaging API changes and ways to implement them appropriately. Deprecated - We will consider a deprecated API element (endpoint and/or parts of an endpoint) to be an element which still provides users with access to its backing functionality and can be used, but will soon be removed completely along with said functionality in an upcoming version. Mandatory endpoint - We will consider an endpoint mandatory if Firecracker cannot operate normally without performing a request to it. Optional endpoint - We will consider an endpoint optional if Firecracker can operate normally without performing a request to it and the functionality behind it is not essential. Mandatory header/field - We will consider a header/field mandatory in an HTTP message if the request will fail without specifying said header/field. Optional header/field - We will consider a header/field optional in an HTTP message if the request will succeeds without specifying said header/field. For the purposes of this document, there are 2 main categories for API changes, namely breaking and non-breaking. A breaking change in the API is a change that makes the API incompatible with the previous version (backwards incompatible). In an effort to avoid a breaking change, we may take the route of deprecation and incrementing the minor version in an effort to preserve backwards compatibility, but breaking changes will always ultimately result in incrementing the major version. Here is a non-exhaustive list of such changes: Adding a new mandatory endpoint/HTTP method. Removing an endpoint/method. Adding a mandatory request header/field. Removing a request header/field. Adding a mandatory response field. Removing a response header/field. A change in the API is not a breaking change if the version resulting from it is compatible with the previous one (backwards compatible). The outcome of a non-breaking change should always include incrementing the minor version but must not lead to incrementing the major version by itself. Here is a non-exhaustive list of such changes: Deprecating an endpoint/method/field. Adding a new optional endpoint/method. Adding an optional request header/field. Adding a response header. Adding additional valid inputs for fields in API requests. Making mandatory headers/fields optional. Making mandatory endpoints optional. Changing the URI of an endpoint. Changing the metrics output format. API changes result in version increases. As Firecrackers support policy is based on , we will look at API changes from this point of view. Given a version number MAJOR.MINOR.PATCH, increment the: MAJOR version when you make incompatible API changes; MINOR version when you add functionality in a backwards compatible manner; PATCH version when you make backwards compatible bug fixes. *All deprecated endpoints are supported until at least the next major version release, where they may be

We will go through multiple types of API changes and provide ways to ensure we dont break our backwards compatibility promise to our customers. The list is split into categories of components changed. Entire endpoints Adding an optional endpoint with new functionality - Increment minor version. Adding a command line parameter - Increment minor version. Removing an endpoint - Deprecate endpoint and increment minor version Remove endpoint when incrementing major version. Adding a mandatory endpoint - Increment major version. Request Adding an optional header/field - Increment minor version. Renaming a header/field - Accept both names and deprecate the old one Remove old name when incrementing major version. Removing a header/field - Make said header/field optional Remove header/field when incrementing major version. Changing the URI of an endpoint - Redirect the old endpoint to the new one and deprecate the old one Remove old endpoint when incrementing major version. Adding a mandatory header/field - Increment major version. Response Adding a header/field - Create a new, separate endpoint with the changes and deprecate the old one Remove old endpoint when incrementing major version. Removing a header/field - Create a new, separate endpoint with the changes and deprecate the old one Remove old endpoint when incrementing major version. Command line parameter Renaming a command line parameter - Accept both names and deprecate the old one Remove old name when incrementing major version. Changing expected value taken by a command line parameter - Accept both names and deprecate the old one Remove old name when incrementing major version. In case the outlined solution for your case is not feasible (e.g. because of security concerns), break the glass and increment the major version. As outlined in the diagram above, sometimes we have to deprecate endpoints partially or entirely. In this section we will go through different situations where we have to deprecate something and ways of avoiding common pitfalls when doing so. Some paths in the flowchart above lead to deprecation. Based on the initial conditions, there are 2 major cases where we need to deprecate an endpoint: Changing an existing endpoint Often happens because directly changing the endpoint would be a breaking change. We usually create a clone of the old endpoint we want to deprecate and make the necessary changes to it. We usually expose both endpoints in the next minor version while marking the old one as deprecated. The old endpoint retains its previous name. When naming the new endpoint: for HTTP endpoints we follow a per-endpoint versioning scheme; in cases where we cant find a fitting name for the new endpoint, the simplest way forward is to take the old URI and append `/v2` to it. for command line endpoints, we can usually find a different name for the new

Deprecating an endpoint without adding a replacement to it Often happens when we want to phase out a certain feature or functionality, but doing so immediately would be a breaking change. We just mark the endpoint as deprecated. Make sure that any changes you make in the code are also reflected in the swagger specification. Some tips: There is nothing in the swagger file that shows whether an endpoint is mandatory or optional, its all code logic. Mandatory fields in a request or response body are marked with `required: true` in the swagger definition. All other fields are optional. If you need to redirect an endpoint, you have to clone the old one under the new URI in the swagger specification. When marking: an HTTP endpoint as deprecated: Add a comment for the parsing function of the endpoint stating that it is deprecated. Log a `warn!` message stating that the user accessed a deprecated endpoint. Increment the `deprecatedHttpApi` metric. Include the `Deprecated` header in the response. a header field in an HTTP endpoint as deprecated: Add a comment in the parsing function where we check the presence of the header stating that it is deprecated. If the header is present, log a `warn!` message stating that the user used a deprecated field. Increment the `deprecatedHttpApi` metric. Include the Deprecated header in the response. a command line parameter as deprecated: Mention it is deprecated in the help message of the parameter in the argument parser. Add it in the `warndeprecatedparameters` function where we log it and increment the `deprecatedCmdLineApi` metric. When doing a major release, the API can have breaking changes. This is the _only time_ where we can safely remove deprecated elements of the API. To remove a deprecated element of the API: Remove the associated functionality from the codebase (usually in `vmm` or `mmds`); Remove the parsing logic in `api_server`; Remove any unit and integration tests associated with this element. In this guide we set out to remove the `vsock_id` field in `PUT`s on `/vsock`. This was implemented in and we will go step by step through the changes in order to understand the process of changing something in the Firecracker API. We go through the flowchart; we want to remove a field in the body of a HTTP request. So we follow the flowchart like this: Change an existing endpoint Request Remove header or field Make it optional Deprecate Increment minor version. Now that we know we need to make the field optional and deprecate it, its time for the code changes (reference implementation in ). We go to the function in `api_server/src/requests` which is responsible for parsing this request, which is `parseputvsock` in this case, and do the

We find the associated `vmmconfig` struct which `serdejson` uses for deserialization, in this case `VsockDeviceConfig`. In the struct referenced above, we make the parameter optional by encapsulating it in an `Option` with `#[serde(default)]` and `#[serde(skipserializingif = "Option::is_none")]` so that we dont break existing implementations, but we follow the new, desired usage of the endpoint. After deserializing the body of the request into the struct, we check for the existence of the field we want to deprecate, in this case by calling `vsockcfg.vsockid.is_some()`. If the field is there, we must mark this request as being deprecated, so we craft a deprecation message (`"PUT /vsock: vsock_id field is deprecated."`) and increment the deprecated HTTP API metric (`METRICS.deprecatedapi.deprecatedhttpapicalls.inc()`). We create a new `ParsedRequest` where, if we marked the request as deprecated, we append the deprecation message into its `parsing_info` structure, in this case by calling `parsedreq.parsinginfo().appenddeprecationmessage(msg)`. Dont forget to comment your code! Comments should reflect what is deprecated and clearly describe the code paths where you handle the deprecation case. Add a unit test where you test your new code paths. Fix all other failing unit tests. Update the swagger file to reflect the change, in this case by removing the `vsock_id` field from the required parameter list in the `Vsock` definition and adding a description to it stating that it is deprecated since the current version. Update any relevant documentation. We update the python integration tests to reflect the change (reference implementation in ). We refactor the relevant `tests/integrationtests/functional/testapi.py` test to use the artifact model instead of the fixture one. If the test already uses the artifact model, you can skip this step. We make sure to run the test with the current build, as well as with future Firecracker versions by specifying the unreleased version in the `min_version` parameter of `artifacts.firecrackers()`. We do this in order to ensure that, when we create patch releases on older branches, we test the API with future binaries to enforce backwards compatibility. Disclaimer: This test will fail when running with the binary artifact fetched from S3 until you update the binary there with your current build. You should only do this once your PR has all necessary approves and this test is the last thing keeping it from getting merged. We check that, when the deprecated field is present in the request, the `Deprecation` header is also present in the response by asserting `response.headers['deprecation']`. We do not also check that the header is not present when the field is not present because, in a future version, some other field may be deprecated in the same request and would return the header anyway, resulting in a fail in our test when it shouldnt. Fix all other failing integration tests.