Package: valconfig
Validating configSimple, extendable configuration objectsQuick startInstallationAs apackage:pipinstallvalconfigOr asinlined source:Copy thevalconfig.pyfile into your project and installPydantic.Example usage# config.pyfromvalconfigimportValConfigfrompydanticimportHttpUrlfromscityping.numpyimportArrayclassConfig(ValConfig):n_units:int=5connectivites:Array[float,2]# 2D array of floatsurl:HttpUrl="example.com"config=Config()For more detailed usage, see thedocumentation.MotivationMany types of projects make use of some system to set configuration options,
and while the type of project may influence that system, its basic requirements
are pretty much always the same:It should be easy to read.It should be easy to extend.Validating configcame about to fulfill these needs for scientific programming,
which is a bit of an extreme case: On the one hand, scientists are not professional programmers,
so the system needs to be so simple that it needs no documentation at all.
On the other hand, they are also constantly developing new methods, and so
also need to be able to add new configuration options and new variable types.
Moreover, most users of lab-scale projects are also possible contributors to those projects.We therefore need to allow users to set local configuration options without breaking others’ config, and without requiring anything more complicated than angit push.(Local configuration may include whether to use a GPU, where to save data/figures, MPI flags, etc.)Thus we needed a way toSet version-controlled defaults, that are packaged with the repository.Make it easy to know what the configurable options are.Make it easy to change those options, without touching the defaults.In addition, projects often depend on other projects, so we need a way to settheiroptions as well.Finally, not all parameters are strings or simple numerical values. Some might need to
be cast to NumPy arrays; others might need to be pre-processed, or checked for
consistency with other parameters, or simply inferred from other parameters.
These are all roles that conceptually belong in a configuration parser, not the core parts of our program.
As scientist programmers, we already have the bad habit of mixing up our programming logic into a big ball of spaghetti – let’s give ourselves one less reason to do so.This adds two additional desiderata:Support for composition: Have one configuration for multiple packages.Validation of values based on provided type information.What aValidatingConfigprovidesA mechanism to define simple lists of configuration options withzero boilerplate:
just list the parameter names, their types, and optionally their default.Built-in validationprovided byPydantic.Additional validation forscientific types, when used in conjunction withscitying.An optional mechanism to autogenerate a file for users’ local configuration,
with usage instructions embedded in the file.The ability tocomposeconfiguration objects from multiple packages into a
single central configuration object –even when those packages don’t useValidatingConfig. This ensures all packages have access to a single source of truth.The ability toextendthe functionality of a config object with all the
functionality Python has to offer: read environment variables withos.getenv,
get command line parameters withsys.argv, etc.Other packages typically need to add features like these because their config objects are
more limited. We avoid this by definingconfigwith a standard class.
This keeps this package lean, and your code simple.Other featuresZero onboarding:Uses only standard Python: No custom function calls, no boilerplate.
If you know modern Python, you already know how to use this module.Just define aConfigclass with the parameter names and their types.Use theConfigclass as any other Python class.Define your own custom types with eitherPydanticorScityping. [^new-types]Pydanticprovidesvalidatorswhich can be used to add per-field
validation and pre-processing. These are defined with plain Python,
not a restricted minilanguage or a sanitizedeval.Use standardpropertiesto define computed (inferred) values.Use any file format.The default uses stdlib’sconfigparser, but you can override the method
used to read files.Relative paths in config files are correctly resolved.Automatic discovery of the local configuration file.Just set the class variable__local_config_filename__, andValConfigwill recurse up the directory tree until it finds a matching file.This is especially convenient to accommodate multiple users who organize
their work differently.Hierarchical: Organize your parameters into categories by defining nested classes.fromvalconfigimportValConfigclassConfig(ValConfig):classfigures:width:floatformat:strclasscurves:colormap:str|list[str]classheatmaps:colormap:str|list[str]classrun:cache:boolclassmodel:n_units:intComposableWant multiple config objects for each of your project’s subpackages ?
Just import them.Want to combine them into a single root config file ?
Just import them.fromvalconfigimportValConfigfrom.subpkg1importconfigasconfig1from.subpkg2importconfigasconfig2classConfig(ValConfig):pkg1:config1pkg2:config2Does your project depend on another ?
Define the parameters you need to modify and add a function which updates
the config with those parameters.fromvalconfigimportValConfigfromother_packageimportconfigasOtherConfigclassConfig(ValConfig):# List the parameters you want to modifyother_package_option1:intother_package_option2:float# Include the 3rd party configother_package:OtherConfig@validator("other_package")defupdate_other_package_options(cls,other_config,values):# The exact details here will depend on the 3rd party packageother_config.opt1=values["other_package_option1"]other_config.opt2=values["other_package_option2"][^new-types]:Scitypingwas developed as an extension ofPydanticto allow
the use of (abstract) base classes in type definitions, for example defining
a field of typeModelwhich accepts any subclass ofModel. (In plain
Pydantic values are alwayscoercedto the target type.) Whether it is best
to define new types with eitherScitypingorPydanticlargely depends on
whether this use as abstract classes is needed.In relation to other projectsWhat’s wrong with the gazillion other config file projects out there ?
Why not use one of those ?In short, because we can now do in 200 lines of code what used to take thousands.
Most existing packages for config files were developed many years ago, before
the standardization of type annotations. The can all perform the basic task of
converting a file to a Python dictionary of strings, but fulfilling all of our
aforementioned desiderata was difficult without creating a bloated package.
Understandly therefore, they focused on the features required by their own use
cases, which means that I found them all unsatisfactory in some respect:Some introduce a new API with custom functions or decorators to define parameters. This makes it more difficult to learn and extend. (vyper-config,hydra)Many provide no validation (one-config,config2).When validations functions are provided, the target type is often not specified
by the config object, but in the calling code – if your configuration library is allows to define
thenamebut not thetypeof the parameters, that’s only 50% of the information and 20% of the work.
(configparser,vyper-config)For the examples I know which provides validation at the config level,
the set of supported types is very limited and basically hard-coded. (OmegaConf,CFG API)Some approaches even define their own file formats, substantially raising the barrier to adoption. (CFG API)The packageconfigobjis probably the most closely aligned with our goals: it provides a simple, declarative format for both parameter and types specification, as well as key features like hierarchical parameters. (But not, to my knowledge, the configuration of subpackages.) It is also mature, which unfortunately means that it pre-dates widespread use of validation libraries and therefore must package its own custom validation library.On the other hand,configobjmight make a great substitute to get around some of the limitations of the builtinconfigparserwith regards to reading files.A simple implementationWith the standardization of type annotations in Python 3.6–3.8, and the availability of classes likePydantic’sBaseModel, defining classes with validation
logic has become a breeze, and converting aBaseModelinto a full-featured
config parser basically only needs two things:functionality for reading values from files;ability to compose configuration classes.In effect therefore,ValConfigis just a subclass ofBaseModelwith some
simple methods for reading config files. The biggest difference is that each
subclass ofValConfigis made asingleton. We use this pattern for exactly
the same reasons asone-config: it
solves a host of corner cases, makes it trivial to support composing configs,
and ensures that config objects are a single source of truth.By relying on Pydantic,ValConfigcan be extremely lean, while still providing
functionality that is on-par with the much heavier packages listed above.
Pydantic itself is highly mature and actively maintained.TheValConfigclass and all its utility functions clock in at less than
500 lines of code and are contained in a single, easily maintained module. If
desired, the module can even be included as part of your project’s source
code, thus removingvalconfigas package dependency and giving you
full control over the config object.Finally,using ValConfig does not preclude from using another config file parser.
Indeed, the provided implementation usesconfigparserto parse files mostly
because it is already installed part of the standard library. The execution flow
is basicallyRead config file(s) withconfigparser.Validate withPydantic.To use a different config file parser, just subclassValConfigand override the
the methodread_cfg_file().