Package: schema-overseer-local
Schema Overseer – LocalThis is a local version of Schema Overseer, intended to use in a single repository.For the multi-repository service seeschema-overseer-service.Schema Overseerensures strict adherence to defined data formats and raises an exception in case of attempting to process unsupported input schema.In more technical terms, it is an adapter[^1] between inputs with different schemas and other application components.Why is it important?Data formats evolve over timeDevelopers need to simultaneously support both legacy and new data formatsMismatches between input data format and the corresponding code can lead to unexpected and hard-to-debug runtime errorsAs the number of supported data formats increases, application code often becomes less maintainableFeaturesStraightforward extensibilityStatic analysis checks via type checkingDetailed runtime checksIncoming data validation withpydanticUse Cases and TutorialsMaintain multiple version of external-facing APIManage metadata for different Machine Learning modelsInstallationpipinstallschema-overseer-localQuick StartCreate a fileadapter.pyto define the adapter logic.For quick start we will use single file, but in a real application it's better to usemultiple files.Output. Define the output schema you plan to work with.The output schema could be any object. For the tutorial purpose we will usedataclass. The output schema attributes could be any python objects, including non-serializables. Output could have the same behavior as the original input object, or a completely different one. Here is example of different behavior.@dataclassclassOutput:value:intfunction:CallableRegistry. Create theSchemaRegistryinstance forOutput.schema_registry=SchemaRegistry(Output)Input schemas. Define the input schemas usingpydanticand register them inschema_registry.@schema_registry.add_schemaclassOldInputFormat(BaseModel):value:str@schema_registry.add_schemaclassNewInputFormat(BaseModel):renamed_value:intBuilders. Implement functions to convert each registered input toOutput.Builders require type hinting to link input formats andOutput.@schema_registry.add_builderdefold_builder(data:OldInputFormat)->Output:returnOutput(value=data.value,function=my_function,)@schema_registry.add_builderdefnew_builder(data:NewInputFormat)->Output:returnOutput(value=data.renamed_value,function=my_other_function,)Finally, useschema_registryinside the application to get validated output or handle the exception.schema_registry.setup()# see "Discovery" chapter in documentationdefmy_service(raw_data:dict[str,Any]):try:output=schema_registry.build(source_dict=raw_data)# build output objectexceptBuildErroraserror:raiseMyApplicationError()fromerror# handle the exception# use output objectoutput.function()returnoutput.valueFull quickstart example ishereRun it:gitclonegit@github.com:Schema-Overseer/schema-overseer-local.gitcdschema-overseer-local
poetryinstall
poetryrunpython-mtutorial.quickstart.appUsageUsing multiple Python filesWhile you can define registry, models and builders in one or two files, it is usually a better idea to split them into different files, i.e., Python modules.There are different ways to do the file structure, we recommend one of the following:Minimal — start with this one, when you are still figuring out the best way to workExpanded builders — useful for the case with lots of code for each builderDetached output — useful for the case, when the output is a big or complex entityMinimal├── __init__.py
├── builders.py
├── models
│   ├── __init__.py
│   ├── v1.py
│   ├── v2.py
│   ├── ...
│   └── vN.py
└── registry.pyExpanded builders├── __init__.py
├── builders
│   ├── __init__.py
│   ├── v1.py
│   ├── v2.py
│   ├── ...
│   └── vN.py
├── models
│   ├── __init__.py
│   ├── v1.py
│   ├── v2.py
│   ├── ...
│   └── vN.py
└── registry.pyDetached output├── __init__.py
├── builders.py
├── models
│   ├── __init__.py
│   ├── v1.py
│   ├── v2.py
│   ├── ...
│   └── vN.py
├── output.py
└── registry.pyModels (i.e., input data formats) are decoupled first for two reasons:If models contain inner models inside, it would be harder to distinguish between inner models for different root models. (seeQ)If you transition toschema-overseer-service, the models are sourced from the outside of your code, so this split will come naturally.Load modules automatically[!NOTE]
Python will not load modules automatically, unless they are explicitly imported.SchemaRegistryhas adiscovery_paths: Sequence[str]argument to load all required models.Specified modules and packages will be loaded atSchemaRegistry.setup().Definition (SchemaRegistry(...)) is decoupled with loading (SchemaRegistry.setup()) to prevent cycle imports, that's why callingsetup()is required.Argumentdiscovery_pathstakes a sequence of strings in the absolute import format. Entries could be either python modules (single files) or python packages (folder with__init__.pyand other*.pyfiles inside)For example, this will work for minimal option, mentioned above:schema_registry=SchemaRegistry(Output,discovery_paths=['example_project.payload.models',# loaded as package'example_project.payload.builders',# loaded as module],)Runtime safety and strict self-checksIn addition to static type hint checks,schema-overseer-localperforms runtime checks to ensure:Each registered model has only one corresponding builder.All builders have a proper call signature, which includes:One argument for the input dataNo additional non-default argumentsAll builders have proper type hintsAdditional runtime checks:If set tovalidate_output=True(the default isFalse), it verifies whether the builder returns an object of the annotated type using pydantic.By default,schema-overseer-localselects the builder from the first valid schema. However, ifcheck_for_single_valid_schema=Trueis enabled, it ensures only one schema is valid for the input data.If multiple schemas are found to be valid, aMultipleValidSchemasErrorwill be raised.Object as a sourceSchemaRegistry.build()method operates in two modes:Using dict-like objects as inputs and extracting fields with__getitem__Usebuild(source_dict=...)for this optionUsing objects with data as attributes and extracting fields withgetattrUsebuild(source_object=...)for this optionsource_dictandsource_objectare mutually exclusive.Use one of the input schema as outputTODOFAQQ: Why is this project exists? Isn't it too much overhead for such a simple task?A:It depends on the scale of the different formats you need to support. In case of a few formats to support,schema-overseer-localwould be an overhead indeed. But in the projects with lots of different formats, such extensive adapter layer could be helpful. Another goal ofschema-overseer-localis to serve as a fast and simple introduction toschema-overseer-servicefor sophisticated use cases with multiple teams and repositories to work with.Q: How is this project better than an adapter I can code quickly myself?A:schema-overseer-localhas three important benefits:
- it provides type checking;
- it has very detailed runtime checks;
- and it is easily extensible.Q: Why I have to use type hinting in builders?A:schema-overseer-localuses the same pattern aspydanticandFastAPIfor input and output validation in both runtime and static analysis. It provides an extra layer of defense against code errors. Even if your code is not entirely correctly typed or not checked with static analysis tools likemypy, the data is still validated.Q: Should I re-use inner pydantic models in different data formats?Code exampleclassInnerModel(BaseModel):value:intclassInputFormatV1(BaseModel):inner:InnerModelclassInnerModelV2(BaseModel):value:intclassInputFormatV2(BaseModel):inner:InnerModelV2# or re-use InnerModel?A:Not really. While it might be tempting to adhere to the DRY[^2] principle in this context, it's generally a better approach to fully separate nested pydantic models into distinct modules, avoiding their reuse even if they are identical.The primary rationale is future code maintainability: tracking modifications in reused models can be challenging, and the introduction of a new format version could require changes to the inner model, which would then demand separation regardless.[^1]:Adapter pattern[^2]:Don't repeat yourself