Package: universi
UniversiModernStripe-likeAPI versioning in FastAPIInstallationpipinstalluniversiWho is this for?Universi allows you to support a single version of your code, auto-generating the code/routes for older versions. You keep versioning encapsulated in small and independent "version change" modules while your business logic knows nothing about versioning.Its approach will be useful if you want to:Support many API versions for a long timeEffortlessly backport features and bugfixes to all of your versionsOtherwise, more conventional methods of API versioning may be preferable.TutorialThis guide provides a step-by-step tutorial for setting up automatic API versioning using Universi library. I will illustrate this with an example of a User API, where we will be implementing changes to a User's address.A dummy setupHere is an initial API setup where the User has a single address. We will be implementing two routes - one for creating a user and another for retrieving user details. We'll be using "int" for ID for simplicity.The first API you come up with usually doesn't require more than one address -- why bother?So we create our file with schemas:frompydanticimportBaseModelclassUserCreateRequest(BaseModel):address:strclassUserResource(BaseModel):id:intaddress:strAnd we create our file with routes:fromversions.latest.usersimportUserCreateRequest,UserResourcefromuniversiimportVersionedAPIRouterrouter=VersionedAPIRouter()@router.post("/users",response_model=UserResource)asyncdefcreate_user(payload:UserCreateRequest):return{"id":83,"address":payload.address,}@router.get("/users/{user_id}",response_model=UserResource)asyncdefget_user(user_id:int):return{"id":user_id,"address":"123 Example St",}Turning address into a listDuring our development, we have realized that the initial API design was wrong and that addresses should have always been a list because the user wants to have multiple addresses to choose from so now we have to change the type of the "address" field to the list of strings.frompydanticimportBaseModelfrompydanticimportFieldclassUserCreateRequest(BaseModel):addresses:list[str]=Field(min_items=1)classUserResource(BaseModel):id:intaddresses:list[str]@router.post("/users",response_model=UserResource)asyncdefcreate_user(payload:UserCreateRequest):return{"id":83,"addresses":payload.addresses,}@router.get("/users/{user_id}",response_model=UserResource)asyncdefget_user(user_id:int):return{"id":user_id,"addresses":["123 Example St","456 Main St"],}But every user of ours will now have their API integration broken. To prevent that, we have to introduce API versioning. There aren't many methods of doing that. Most of them force you to either duplicate your schemas, your endpoints, or your entire app instance. And it makes sense, really: duplication is the only way to make sure that you will not break old versions with your new versions; the bigger the piece you duplicating -- the safer. Of course, the safest being duplicating the entire app instance and even having a separate database. But that is expensive and makes it either impossible to make breaking changes often or to support many versions. As a result, either you need infinite resources, very long development cycles, or your users will need to often migrate from version to version.Stripe has come upwith a solution: let's have one latest app version whose responses get migrated to older versions and let's describe changes between these versions using migrations. This approach allows them to keep versions foryearswithout dropping them. Obviously, each breaking change is still bad and each version still makes our system more complex and expensive, but their approach gives us a chance to minimize that. Additionally, it allows us backport features and bugfixes to older versions. However, you will also be backporting bugs, which is a sad consequence of eliminating duplication.Universi is heavily inspired by this approach so let's continue our tutorial and now try to combine the two versions we created using versioning.Creating the MigrationWe need to create a migration to handle changes between these versions. For every endpoint whoseresponse_modelisUserResource, this migration will convert the list of addresses back to a single address when migrating to the previous version. Yes, migratingback: you might be used to database migrations where we write upgrade migration and downgrade migration but here our goal is to have an app of latest version and to describe what older versions looked like in comparison to it. That way the old versions are frozen in migrations and you canalmostsafely forget about them.frompydanticimportFieldfromuniversi.structureimport(schema,VersionChange,convert_response_to_previous_version_for,)classChangeAddressToList(VersionChange):description=("Change user address to a list of strings to ""allow the user to specify multiple addresses")instructions_to_migrate_to_previous_version=[# You should use schema inheritance if you don't want to repeat yourself in such casesschema(UserCreateRequest).field("addresses").didnt_exist,schema(UserCreateRequest).field("address").existed_as(type=str,info=Field()),schema(UserResource).field("addresses").didnt_exist,schema(UserResource).field("address").existed_as(type=str,info=Field()),]@convert_response_to_previous_version_for(UserResource)defchange_addresses_to_single_item(payload:dict[str,Any])->None:data["address"]=data.pop("addresses")[0]# Note that this approach is outdated and it is now recommended to use request migrations@schema(UserCreateRequest).property("addresses").wasdefaddresses_property(parsed_schema):return[parsed_schema.address]See how we are popping the first address from the list? This is only guaranteed to be possible because we specified earlier thatmin_itemsforaddressesmust be1. If we didn't, then the user would be able to create a user in a newer version that would be impossible to represent in the older version. I.e. If anyone tried to get that user from the older version, they would get aResponseValidationErrorbecause the user wouldn't have data for a mandatoryaddressfield. You need to always keep in mind tht API versioning is only for versioning yourAPI, your interface. Your versions must still be completely compatible in terms of data. If they are not, then you are versioning your data and you should really go with a separate app instance. Otherwise, your users will have a hard time migrating back and forth between API versions and so many unexpected errors.See how we added theaddressesproperty? This simple instruction will allow us to useaddresseseven from the old schema, which means that our api route will not need to know anything about versioning. The main goal of universi is to shift the logic of versioning away from your business logic and api endpoints which makes your project easier to navigate and which makes deleting versions a breeze.Grouping Version ChangesFinally, we group the version changes in theVersionBundleclass. This represents the different versions of your API and the changes between them. You can add any "version changes" to any version. For simplicity, let's use versions 2002 and 2001 which means that we had a single address in API in 2001 and added addresses as a list in 2002's version.fromuniversi.structureimportVersion,VersionBundlefromdatetimeimportdatefromcontextvarsimportContextVarapi_version_var=ContextVar("api_version_var")versions=VersionBundle(Version(date(2002,1,1),ChangeAddressToList),Version(date(2001,1,1)),api_version_var=api_version_var,)That's it. You're done with describing things. Now you just gotta ask universi to do the rest for you. We'll need the VersionedAPIRouter we used previously, our API versions, and the module representing the latest versions of our schemas.fromversionsimportlatest,api_version_varfromuniversiimportregenerate_dir_to_all_versions,generate_all_router_versionsregenerate_dir_to_all_versions(latest,versions)router_versions=generate_all_router_versions(router,versions=versions,latest_schemas_module=latest,)api_version_var.set(date(2002,1,1))uvicorn.run(router_versions[date(2002,1,1)])Universi has generated multiple things in this code:Three versions of our schemas: one for each API version and one that includes definitions of unions of all versions for each schema which will be useful when you want to type check that you are using requests of different versions correctly. For example, we'll haveUserCreateRequestdefined there which is aTypeAliaspointing to the union of 2002 version and 2001 version ofUserCreateRequest.Two versions of our API router: one for each API versionYou can now just pick a router by its version and run it separately or use a parent router/app to specify the logic by which you'd like to pick a version. I recommend using a header-based router with version dates as headers. And yes, that's how Stripe does it.Note that universi migrates your response data based on theapi_version_varcontext variable so you must set it with each request.universi.get_universi_dependencydoes that for you automatically on every request based on header value.Obviously, this was just a simple example and universi has a lot more features so if you're interested -- take a look at the reference.ExamplesPlease, seetutorial examplesfor the fully working version of the project above.Important warningsThe goal of Universi is tominimizethe impact of versioning on your business logic. It provides all necessary tools to prevent you fromeverchecking for a concrete version in your code. So please, if you are tempted to check something likeapi_version_var.get() >= date(2022, 11, 11)-- please, take another look intoreferencesection. I am confident that you will find a better solution there.Universi does not include a header-based router like FastAPI. We hope that soon a framework for header-based routing will surface which will allow universi to be a full versioning solution.I ask you to be very detailed in your descriptions for version changes. Spending these 5 extra minutes will potentially save you tens of hours in the future when everybody forgets when, how, and why the version change was made.We migrate responses backwards in versions from the latest version using data migration functions and requests forward in versions until the latest version using properties on pydantic models.Universi doesn't edit your imports when generating schemas so if you make any imports from versioned code to versioned code, I would suggest usingrelative importsto make sure that they will still work as expected after code generation.ReferenceEndpointsNote that the endpoint constructor contains a second argument that describes the methods of the endpoints you would like to edit. If you have two routes for a single endpoint and you put both of their methods into the instruction -- both of them are going to be changed as you would expect.Defining endpoints that didn't exist in new versionsIf you had an endpoint in old version but do not have it in a new one, you must still define it but mark it as deleted.@router.only_exists_in_older_versions@router.get("/my_old_endpoint")asyncdefmy_old_endpoint():...and then define it as existing in one of the older versions:fromuniversi.structureimportVersionChange,endpointclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[endpoint("/my_old_endpoint",["GET"]).existed,]Defining endpoints that didn't exist in old versionsIf you have an endpoint in your new version that must not exist in older versions, you define it as usual and then mark it as "non-existing" in old versions:fromuniversi.structureimportVersionChange,endpointclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[endpoint("/my_new_endpoint",["GET"]).didnt_exist,]Changing endpoint attributesIf you want to change any attribute of your endpoint in a new version, you can return the attribute's value in all older versions like so:fromuniversi.structureimportVersionChange,endpointclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[endpoint("/my_endpoint",["GET"]).had(description="My old description"),]Dealing with endpoint duplicatesSometimes, when you're doing some advanced changes in between versions, you will need to rewrite your endpoint function entirely. So essentially you'd have the following structure:fromfastapi.paramsimportParamfromfastapi.headersimportHeaderfromtypingimportAnnotatedfromuniversiimportVersionedAPIRouterrouter=VersionedAPIRouter()@router.only_exists_in_older_versions@router.get("/users")defget_users_by_name_before_we_started_using_params(user_name:Annotated[str,Header()]):"""Do some logic with user_name"""@router.get("/users")defget_users_by_name(user_name:Annotated[str,Param()]):"""Do some logic with user_name"""As you see, these two functions have the same methods and paths. And when you have many versions, you can have even more functions like these two. So how do we ask universi to restore only one of them and delete the other one?fromuniversi.structureimportVersionChange,endpointclassUseParamsInsteadOfHeadersForUserNameFiltering(VersionChange):description=("Use params instead of headers for user name filtering in GET /users ""because using headers is a bad API practice in such scenarios.")instructions_to_migrate_to_previous_version=[# We don't have to specify the name here because there's only one such deleted endpointendpoint("/users",["GET"]).existed,# We do have to specify the name because we now have two existing endpoints after the instruction aboveendpoint("/users",["GET"],func_name="get_users_by_name").didnt_exist,]So by using a more concretefunc_name, we are capable to distinguish between different functions that affect the same routes.EnumsAdding enum membersNote that adding enum memberscanbe a breaking change unlike adding optional fields to a schema. For example, if I return a list of entities, each of which has some type, and I add a new type -- then my client's code is likely to break.So I suggest adding enum members in new versions as well.fromuniversi.structureimportVersionChange,enumfromenumimportautoclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[enum(my_enum).had(foo="baz",bar=auto()),]Removing enum membersfromuniversi.structureimportVersionChange,enumclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[enum(my_enum).didnt_have("foo","bar"),]SchemasAdd a fieldfrompydanticimportFieldfromuniversi.structureimportVersionChange,schemaclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[schema(MySchema).field("foo").existed_as(type=list[str],info=Field(description="Foo")),]You can also specify any string in place of type:schema(MySchema).field("foo").existed_as(type="AnythingHere")It is often the case that you want to add a type that has not been imported in your schemas yet. You can useimport_fromand optionallyimport_asto do this:schema(MySchema).field("foo").existed_as(type=MyOtherSchema,import_from="..some_module",import_as="Foo")Which will render as:from..some_moduleimportMyOtherSchemaasFoofrompydanticimportBaseModel,FieldclassMySchema(BaseModel):foo:Foo=Field()Remove a fieldfromuniversi.structureimportVersionChange,schemaclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[schema(MySchema).field("foo").didnt_exist,]Change a fieldfromuniversi.structureimportVersionChange,schemaclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[schema(MySchema).field("foo").had(description="Foo"),]Add a propertyfromuniversi.structureimportVersionChange,schemaclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[]@schema(MySchema).property("foo").wasdefany_name_here(parsed_schema):returnparsed_schema.some_other_fieldRemove a propertyfromuniversi.structureimportVersionChange,schemaclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[schema(MySchema).property("foo").didnt_exist,]Rename a schemaIf you wish to rename your schema to make sure that its name is different in openapi.json:fromuniversi.structureimportVersionChange,schemaclassMyChange(VersionChange):description="..."instructions_to_migrate_to_previous_version=[schema(MySchema).had(name="OtherSchema"),]which will replace all references to this schema with the new name.Private fieldsThe perfect Universi workflow consists of migrating all responses to earlier versions and all requests to latest version automatically. However, it is not always easily possible. For example, if there is more data in older requests than in newer requests, how do we push this data into newer request schemas? They don't have any field for it and we cannot add a field because we don't want our API users to see this field.Private attrscome to the rescue. They allow you to define fields that won't appear in openapi, won't be validated, and won't appear in__init__so our API clients won't be able to use them no matter how hard they try. However, now we have a problem:wealso cannot use them to push information from old versions into new versions because these attrs don't appear in__init__. That's why we haveFillablePrivateAttr. They are the same as private attrs except that you can set them in model's__init__automatically. Note thatFillablePrivateAttrMixinis required to use them.frompydanticimportBaseModelfromuniversi.fieldsimportFillablePrivateAttr,FillablePrivateAttrMixinclassUserCreateRequest(FillablePrivateAttrMixin,BaseModel):default_address:str_addresses_to_create:list[str]=FillablePrivateAttr(default_factory=list)UserCreateRequest(default_address="My street",_addresses_to_create=["My address 2","My address 3"])UnionsAs you probably realize, when you have many versions with different request schemas and your business logic receives one of them -- you're in trouble. You could handle them all separately by checking the version of each schema and then using the correct logic for it but universi tries to offer something better.Instead, we take a union of all of our request schemas and write our business logic as if it receives that union. For example, if version 2000 had field "foo" of typestrand then version 2001 changed that field to typeint, then a union of these schemas will have foo asstr | intso your type checker will protect you against incorrect usage. Same goes for added/deleted fields. Obviously, manually importing all your schemas and then taking a union of them is tough, especially if you have many versions, which is why Universi not only generates a directory for each of your versions, but it also generates a "unions" directory that contains unions of all your schemas and enums.For example, if we had a schema namedMySchemaand two versions of it: 2000 and 2001, then the union definition will look like the following:from..latestimportmy_schema_moduleaslatest_my_schema_modulefrom..v2000_01_01importmy_schema_moduleasv2000_01_01_my_schema_modulefrom..v2001_01_01importmy_schema_moduleasv2001_01_01_my_schema_moduleMySchema=(latest_my_schema_module.MySchema|v2000_01_01_my_schema_module.MySchema|v2001_01_01_my_schema_module.MySchema)and you would be able to use it like so:fromsrc.schemas.unions.my_schema_moduleimportMySchemaasyncdefthe_entrypoint_of_my_business_logic(request_payload:MySchema):...Note that this feature only affects type checking and does not affect your functionality. Note also that everyrequest migrationthat affects a specific body schema will automatically make Universi omit the body schema from  that version in unions.Data migrationResponse data migrationAs described in the tutorial, universi can convert your response data into older versions. It does so by running your "migration" functions whenever it encounters a version change:fromuniversi.structureimportVersionChange,convert_response_to_previous_version_forfromtypingimportAnyclassChangeAddressToList(VersionChange):description="..."@convert_response_to_previous_version_for(MyEndpointResponseModel)defchange_addresses_to_single_item(response:ResponseInfo)->None:response.body["address"]=response.body.pop("addresses")[0]It is done by applying a versioning decorator to each endpoint with the givenresponse_modelwhich automatically detects the API version by getting it from thecontextvarand applying all version changes until the selected version in reverse. Note that if the version is not set, then no changes will be applied.Request data migrationfromuniversi.structureimportVersionChange,convert_request_to_next_version_forfromtypingimportAnyfrommy_schemas.latestimportUserCreateRequestclassChangeAddressToList(VersionChange):description="..."@convert_request_to_next_version_for(UserCreateRequest)defchange_addresses_to_single_item(request:RequestInfo)->None:request.body["addresses"]=[request.body.pop("address")]Version changes with side effectsSometimes you will use API versioning to handle a breaking change in yourbusiness logic, not in the schemas themselves. In such cases, it is tempting to add a version check and just follow the new business logic such as:ifapi_version_var.get()>=date(2022,11,11):# do new logic here...In universi, this approach is highly discouraged. It is recommended that you avoid side effects like this at any cost because each one makes your core logic harder to understand. But if you cannot, then I urge you to at least abstract away versions and versioning from your business logic which will make your code much easier to read.To simplify this, universi has a specialVersionChangeWithSideEffectsclass. It makes finding dangerous versions that have side effects much easier and provides a nice abstraction for checking whether we are on a version where these side effects have been applied.As an example, let's use the tutorial section's case with the user and their address. Let's say that we use an external service to check whether user's address is listed in it and return 400 response if it is not. Let's also say that we only added this check in the newest version.fromuniversi.structureimportVersionChangeWithSideEffectsclassUserAddressIsCheckedInExternalService(VersionChangeWithSideEffects):description=("User's address is now checked for existense in an external service. ""If it doesn't exist there, a 400 code is returned.")Then we will have the following check in our business logic:fromsrc.versionsimportversions,UserAddressIsCheckedInExternalServiceasyncdefcreate_user(payload):ifUserAddressIsCheckedInExternalService.is_applied:check_user_address_exists_in_an_external_service(payload.address)...So this change can be contained in any version -- your business logic doesn't know which version it has and shouldn't.API Version header and context variablesUniversi automatically converts your data to a correct version and has "version checks" when dealing with side effects as described inthe section above. It can only do so using a specialcontext variablethat stores the current API version.Useuniversi.get_universi_dependencyto get afastapi.Dependsthat automatically sets this contextvar based on a header name that you pick.You can also set the variable yourself or even pass a different compatible contextvar to youruniversi.VersionBundleconstructor.Similar projectsThe following projects are trying to accomplish similar results with a lot more simplistic functionality.https://github.com/sjkaliski/pinnedhttps://github.com/phillbaker/gateshttps://github.com/lukepolo/laravel-api-migrationshttps://github.com/tomschlick/request-migrationshttps://github.com/keygen-sh/request_migrations