Package: dataclass-binder
Dataclass BinderLibrary to bind TOML data to Python dataclasses in a type-safe way.FeaturesCurrently it has the following properties that might set it apart from other data binding libraries:requires Python 3.10+relies only on dataclasses from the Python standard librarydetailed error messages which mention location, expected data and actual datastrict parsing which considers unknown keys to be errorssupport for durations (timedelta)support for immutable (frozen) dataclassescan bind data from files, I/O streams or pre-parsed dictionariescan generate configuration templates from dataclass definitionsThis library was originally designed for parsing configuration files.
As TOML's data model is very similar to JSON's, adding support for JSON in the future would be an option and would make the library useful for binding HTTP API requests.MaturityThis library is fully type-checked, has unit tests which provide 100% branch coverage and is used in production, so it should be reliable.The API might still change in incompatible ways until the 1.0 release.
In particular the following aspects are subject to change:use of key suffixes fortimedelta: this mechanism doesn't work for arraysthe handling of separators in keys: currently-in TOML is mapped to_in Python and_is forbidden in TOML; most applications seem to accept both-and_in TOML insteadWhy Dataclasses?A typical TOML, JSON or YAML parser returns the parse results as a nested dictionary.
You might wonder why you would want to use a data binding library rather than just getting the values directly from that dictionary.Let's take the following example code for a service that connects to a database using a connection URL configured in a TOML file:importtomllib# or 'tomli' on Python <3.11defread_config()->dict:withopen("config.toml","rb")asf:config=tomllib.load(f)returnconfigdefhandle_request(config:dict)->None:url=config["database-url"]print("connect to database:",url)config=read_config()...handle_request(config)If the configuration is missing adatabase-urlkey or its value is not a string, this service would start up without complaints and then fail when the first requests comes in.
It would be better to instead check the configuration on startup, so let's add code for that:defread_config():withopen("config.toml","rb")asf:config=tomllib.load(f)url=config["database-url"]ifnotisinstance(url,str):raiseTypeError(f"Value for 'database-url' has type '{type(url).__name__}', expected 'str'")returnconfigImagine you have 20 different configurable options: you'd need this code 20 times.Now let's assume that you use a type checker likemypy.
Insideread_config(), the type checker will know thaturlis astr, but if you fetch the same value elsewhere in the code, that information is lost:defhandle_request(config:dict)->None:url=config["database-url"]reveal_type(url)print("connect to database:",url)When you runmypyon this code, it will output 'Revealed type is "Any"'.
Falling back toAnymeans type checking will not be able to find type mismatches and autocomplete in an IDE will not work well either.Declaring the desired type in a dataclass solves both these issues:the type can be verified at runtime before instantiating the dataclasstooling knows the type when you read the value from the dataclassHaving the dataclass as a central and formal place for defining the configuration format is also an advantage.
For example, it enables automatic generation of a documented configuration file template.UsageThedataclass_bindermodule contains theBinderclass which makes it easy to bind TOML data, such as a configuration file, to Pythondataclasses.The binding is a two-step process:instantiate theBinderclass by passing your top-level dataclass as an argumentcall theparse_toml()method, providing the path of the configuration file as its argumentPut together, the code looks like this:importloggingimportsysfrompathlibimportPathfromdataclass_binderimportBinderlogger=logging.getLogger(__name__)if__name__=="__main__":config_file=Path("config.toml")try:config=Binder(Config).parse_toml(config_file)exceptExceptionasex:logger.critical("Error reading configuration file '%s':%s",config_file,ex)sys.exit(1)Binding a Pre-parsed DictionaryIf you don't want to bind the contents of a full file, there is also the option to bind a pre-parsed dictionary instead.
For this, you can use thebind()method on theBinderobject.For example, the following service is configured by one table within a larger TOML configuration file:importtomllib# or 'tomli' on Python <3.11fromdataclass_binderimportBinderwithopen("config.toml","rb")asf:config=tomllib.load(f)service_config=Binder(ServiceConfig).bind(config["service"])To keep these examples short, from now onimportstatements will only be included the first time a particular imported name is used.Basic TypesDataclass fields correspond to TOML keys. In the dataclass, underscores are used as word separators, while dashes are used in the TOML file. Let's configure a service that listens on a TCP port for requests and stores its data in a database, using the following TOML fragment:database-url='postgresql://user:password@host/db'port=8080This configuration can be bound to the following dataclass:fromdataclassesimportdataclass@dataclassclassConfig:database_url:strport:intverbose:boolThefloattype can be used to bind floating point numbers.
Support forDecimalis not there at the moment but would be relatively easy to add, astomllib/tomlihas an option for that.DefaultsFields can be made optional by assigning a default value. UsingNoneas a default value is allowed too:@dataclassclassConfig:verbose:bool=Falsewebhook_url:str|None=NoneIf you want to mix fields with and without defaults in any order, mark the fields as keyword-only:@dataclass(kw_only=True)classConfig:database_url:strverbose:bool=Falseport:intFile System PathsWhile you can of course store file system paths as strings,pathliboffers a nicer API for them.
To take advantage of this, annotate the field asPath:frompathlibimportPath@dataclassclassConfig:log_dir:Pathdefcreate_log_dir(self)->None:self.log_dir.mkdir(parents=True)Dates and TimesTOML handles dates and timestamps as first-class values.
Date, time and date+time TOML values are bound todatetime.date,datetime.timeanddatetime.datetimePython objects respectively.There is also support for time intervals usingdatetime.timedelta:fromdatetimeimporttimedelta@dataclassclassConfig:retry_after:timedeltadelete_after:timedeltaIntervals shorter than a day can be specified using a TOML time value.
Longer intervals are supported by adding an-hours,-days, or-weekssuffix.
Other supported suffixes are-minutes,-seconds,-millisecondsand-microseconds, but these are there for completeness sake and less likely to be useful.
Here is an example TOML fragment corresponding to the dataclass above:retry-after=00:02:30delete-after-days=30CollectionsLists and dictionaries can be used to bind TOML arrays and tables.
If you want to make alistordictoptional, you need to provide a default value via thedefault_factorymechanism as usual, see thedataclasses documentationfor details.fromdataclassesimportdataclass,field@dataclassclassConfig:tags:list[str]=field(default_factory=list)limits:dict[str,int]The dataclass above can be used to bind the following TOML fragment:tags=["production","development"]limits={ram-gb=1,disk-gb=100}An alternative todefault_factoryis to use a homogeneous (single element type) tuple:@dataclassclassConfig:tags:tuple[str,...]=()limits:dict[str,int]Heterogeneous tuples are supported too: for exampletuple[str, bool]binds a TOML array that must always have a string as its first element and a Boolean as its second and last element.
It is generally clearer though to define a separate dataclass when you need more than one value to configure something:@dataclassclassWebhook:url:strtoken:str@dataclassclassConfig:webhooks:tuple[Webhook,...]=()The extra keys (urlandtokenin this example) provide the clarity:webhooks=[{url="https://host1/notify",token="12345"},{url="https://host2/hook",token="frperg"}]TOML's array-of-tables syntax can make this configuration a bit easier on the eyes:[[webhooks]]url="https://host1/notify"token="12345"[[webhooks]]url="https://host2/hook"token="frperg"Always define additional dataclasses at the module level in your Python code: if the class is for example defined inside a function, theBinderconstructor will not be able to find it.Untyped DataSometimes the full structure of the data you want to bind is either too complex or too much in flux to be worth fully annotating.
In such a situation, you can usetyping.Anyas the annotation to simply capture the output of Python's TOML parser without type-checking it.In the following example, a service uses the Python standard library logging implementation, configured using theconfiguration dictionary schema:importlogging.configfromdataclassesimportdataclassfromtypingimportAnyfromdataclass_binderimportBinder@dataclassclassConfig:database_url:strlogging:Anydefrun(url:str)->None:logging.info("Service starting")if__name__=="__main__":config=Binder[Config].parse_toml("service.toml")logging.config.dictConfig(config.logging)run(config.database_url)Theservice.tomlconfiguration file for this service could look like this:database-url='postgresql://user:password@host/db'[logging]version=1[logging.root]level='INFO'handlers=['file'][logging.handlers.file]class='logging.handlers.RotatingFileHandler'filename='service.log'formatter='simple'[logging.formatters.simple]format='%(asctime)s %(name)s %(levelname)s %(message)s'PluginsTo select plugins to activate, you can bind Python classes or modules usingtype[BaseClass]andtypes.ModuleTypeannotations respectively:fromdataclassesimportdataclass,fieldfromtypesimportModuleTypefromsupertool.pluginsimportPostProcessor@dataclassclassPluginConfig:postprocessors=tuple[type[PostProcessor],...]=()modules:dict[str,ModuleType]=field(default_factory=dict)In the TOML, you specify Python classes or modules using their fully qualified names:postprocessors=["supertool_addons.reporters.JSONReport"]modules={lint="supertool_addons.linter"}There is no mechanism yet to add configuration to be used by the plugins.ImmutableIf you prefer immutable configuration objects, you can achieve that using thefrozenargument of thedataclassdecorator and using abstract collection types in the annotations. For example, the following dataclass will be instantiated with atupleobject fortagsand an immutable dictionary view forlimits:fromcollections.abcimportMapping,Sequence@dataclass(frozen=True)classConfig:tags:Sequence[str]=()limits:Mapping[str,int]Layered BindingBindercan be instantiated from a dataclass object rather than the dataclass itself.
The dataclass object will provide new default values when binding data to it.
This can be used to implement a layered configuration parsing mechanism, where there is a default configuration that can be customized using a system-wide configuration file and/or a per-user configuration file:config=Config()ifsystem_config_path.exists():config=Binder(config).parse_toml(system_config_path)ifuser_config_path.exists():config=Binder(config).parse_toml(user_config_path)Later layers can override individual fields in nested dataclasses, allowing fine-grained configuration merging, but collections are replaced whole instead of merged.Generating a Configuration TemplateTo provide users with a starting point for configuring your application/service, you can automatically generate a configuration template from the information in the dataclass.For example, when the following dataclass defines your configuration:@dataclassclassConfig:database_url:str"""The URL of the database to connect to."""port:int=12345"""TCP port on which to accept connections."""You can generate a template configuration file using:fromdataclass_binderimportBinderforlineinBinder(Config).format_toml_template():print(line)Which will print:# The URL of the database to connect to.# Mandatory.database-url='???'# TCP port on which to accept connections.# Default:# port = 12345It is also possible to provide placeholder values by passing a dataclass instance rather than the dataclass itself toformat_toml_template():TEMPLATE=Config(database_url="postgresql://<username>:<password>@<hostname>/<database name>",port=8080,)forlineinBinder(TEMPLATE).format_toml_template():print(line)Which will print:# The URL of the database to connect to.# Mandatory.database-url='postgresql://<username>:<password>@<hostname>/<database name>'# TCP port on which to accept connections.# Default:# port = 12345port=8080Generating a Compact Configuration FileIf you want to generate a fully populated configuration, you can use theformat_toml()method.
Compared to the template formatting, this leaves out optional parts for which no data has been provided.For example, when the following dataclass defines your configuration:@dataclassclassConfig:path:str"""Path of input file."""verbose:bool=False"""Be extra verbose in logging."""options:dict[str,Any]=field(default_factory=dict)"""Various named options that are passed on to tool XYZ."""This code generates a populated configuration file:config=Config(path="./data")withopen("config.toml","w")asout:forlineinBinder(config).format_toml():print(line,file=out)With the contents ofconfig.tomlcontaining only thepathfield:# Path of input file.path='./data'TroubleshootingFinally, a troubleshooting tip: instead of the fullBinder(Config).parse_toml(), first try to execute onlyBinder(Config).
If that fails, the problem is in the dataclass definitions.
If that succeeds, but theparse_toml()call fails, the problem is that the TOML file does not match the format defined in the dataclasses.Development EnvironmentPoetryis used to set up a virtual environment with all the dependencies and development tools that you need:$ cd dataclass-binder
$ poetry installYou can activate a shell which contains the development tools in its search path:$ poetry shellWe recommend setting up pre-commit hooks for Git in thedataclass-binderwork area.
These hooks automatically run a few simple checks and cleanups when you create a new commit.
After you first set up your virtual environment with Poetry, run this command to install the pre-commit hooks:$ pre-commit installRelease ProcedureVerify that CI passes on the branch that you want to release (typicallymain)Create a release on the GitHub web interface; name the tagv<major>.<minor>.<patchlevel>After publishing the release on GitHub, the package will be built and published on PyPI automatically via ActionsDeprecationsBinder SpecializationPrior to version 0.2.0, theBinderclass was specialized using a type argument (Binder[Config]) rather than instantiation (Binder(config)). The old syntax is still supported for now, but the backwards compatibility might be removed in a minor release prior to 1.0 if it becomes a maintenance burden, so please update your code.Template FormattingIn version 0.3.0, the functionformat_template()has been replaced by the methodBinder.format_toml_template(). The old function is still available for now.Changelog0.1.0 - 2023-02-21:First open source release; thanks to my employerProtixfor making this possible0.1.1 - 2023-02-22:RelaxBinder.bind()argument type toMapping(#3)0.1.2 - 2023-03-03:Fixget()and[]on object bound to read-only mapping (#6)0.1.3 - 2023-03-05:Ignore dataclass fields withinit=False(#2)0.2.0 - 2023-06-26:InstantiateBinderinstead of specializing it (#14)Supporttyping.Anyas a field annotation (#10)Fix crash informat_template()on optional fields with non-string annotations (#16)0.3.0 - 2023-07-13:Replaceformat_template()function byBinder.format_toml_template()method (#23)Format nested dataclasses as TOML tables (#25)Format untyped mappings and sequences as TOML tables (#27)Fix formatting ofinit=Falsefield in nested dataclasses (#22)Fix annotation evaluation on inherited dataclasses (#21)0.3.1 - 2023-07-17:Generate template in depth-first order (#28)Fix binder creation and formatting for recursive dataclasses (#28)0.3.2 - 2023-07-27:Document fields with adefault_factoryas optional in template (#35)Omit values from template that are formatted equally to the default (#36)Require fields withNonein their annotation to haveNoneas their default (#37)0.3.3 - 2023-07-31:AddBinder.format_toml()method to generate more compact TOML that excludes unused optional parts (#38)0.3.4 - 2023-08-18:Supportpathlib.Pathas a field annotation (#40)