Package: confection
Confection: The sweetest config system for Pythonconfection:candy: is a lightweight library that offers aconfiguration
systemletting you conveniently describe arbitrary trees of objects.Configuration is a huge challenge for machine-learning code because you may want
to expose almost any detail of any function as a hyperparameter. The setting you
want to expose might be arbitrarily far down in your call stack, so it might
need to pass all the way through the CLI or REST API, through any number of
intermediate functions, affecting the interface of everything along the way. And
then once those settings are added, they become hard to remove later. Default
values also become hard to change without breaking backwards compatibility.To solve this problem,confectionoffers a config system that lets you easily
describe arbitrary trees of objects. The objects can be created via function
calls you register using a simple decorator syntax. You can even version the
functions you create, allowing you to make improvements without breaking
backwards compatibility. The most similar config system we’re aware of isGin, which uses a similar syntax, and
also allows you to link the configuration system to functions in your code using
a decorator.confection's config system is simpler and emphasizes a different
workflow via a subset of Gin’s functionality.⏳ Installationpipinstallconfectioncondainstall-cconda-forgeconfection👩‍💻 UsageThe configuration system parses a.cfgfile like[training]patience=10dropout=0.2use_vectors=false[training.logging]level="INFO"[nlp]# This uses the value of training.use_vectorsuse_vectors=${training.use_vectors}lang="en"and resolves it to aDict:{"training":{"patience":10,"dropout":0.2,"use_vectors":false,"logging":{"level":"INFO"}},"nlp":{"use_vectors":false,"lang":"en"}}The config is divided into sections, with the section name in square brackets –
for example,[training]. Within the sections, config values can be assigned to
keys using=. Values can also be referenced from other sections using the dot
notation and placeholders indicated by the dollar sign and curly braces. For
example,${training.use_vectors}will receive the value of use_vectors in the
training block. This is useful for settings that are shared across components.The config format has three main differences from Python’s built-inconfigparser:JSON-formatted values.confectionpasses all values throughjson.loadsto
interpret them. You can use atomic values like strings, floats, integers or
booleans, or you can use complex objects such as lists or maps.Structured sections.confectionuses a dot notation to build nested
sections. If you have a section named[section.subsection],confectionwill parse that into a nested structure, placing subsection within section.References to registry functions. If a key starts with@,confectionwill
interpret its value as the name of a function registry, load the function
registered for that name and pass in the rest of the block as arguments. If
type hints are available on the function, the argument values (and return
value of the function) will be validated against them. This lets you express
complex configurations, like a training pipeline wherebatch_sizeis
populated by a function that yields floats.There’s no pre-defined scheme you have to follow; how you set up the top-level
sections is up to you. At the end of it, you’ll receive a dictionary with the
values that you can use in your script – whether it’s complete initialized
functions, or just basic settings.For instance, let’s say you want to define a new optimizer. You'd define its
arguments inconfig.cfglike so:[optimizer]@optimizers="my_cool_optimizer.v1"learn_rate=0.001gamma=1e-8To load and parse this configuration using acatalogueregistry (installcatalogueseparately):importdataclassesfromtypingimportUnion,Iterableimportcataloguefromconfectionimportregistry,Config# Create a new registry.registry.optimizers=catalogue.create("confection","optimizers",entry_points=False)# Define a dummy optimizer class.@dataclasses.dataclassclassMyCoolOptimizer:learn_rate:floatgamma:float@registry.optimizers.register("my_cool_optimizer.v1")defmake_my_optimizer(learn_rate:Union[float,Iterable[float]],gamma:float):returnMyCoolOptimizer(learn_rate,gamma)# Load the config file from disk, resolve it and fetch the instantiated optimizer object.config=Config().from_disk("./config.cfg")resolved=registry.resolve(config)optimizer=resolved["optimizer"]# MyCoolOptimizer(learn_rate=0.001, gamma=1e-08)⚠️ Caution: Type-checkers such asmypywill mark adding new attributes toregistrythis way - i. e.registry.new_attr = ...- as errors. This is because a new attribute is added to the class after initialization. If
you are using typecheckers, you can either ignore this (e. g. with# type: ignoreformypy) or use a typesafe
alternative: instead ofregistry.new_attr = ..., usesetattr(registry, "new_attr", ...).Under the hood,confectionwill look up the"my_cool_optimizer.v1"function
in the "optimizers" registry and then call it with the argumentslearn_rateandgamma. If the function has type annotations, it will also validate the
input. For instance, iflearn_rateis annotated as a float and the config
defines a string,confectionwill raise an error.The Thinc documentation offers further information on the configuration system:recursive blocksdefining variable positional argumentsusing interpolationusing custom registriesadvanced type annotations with Pydanticusing base schemasfilling a configuration with defaults🎛 APIclassConfigThis class holds the model and trainingconfigurationand can load and save the
INI-style configuration format from/to a string, file or bytes. TheConfigclass is a subclass ofdictand uses Python’sConfigParserunder the hood.methodConfig.__init__Initialize a newConfigobject with optional data.fromconfectionimportConfigconfig=Config({"training":{"patience":10,"dropout":0.2}})ArgumentTypeDescriptiondataOptional[Union[Dict[str, Any], Config]]Optional data to initialize the config with.section_orderOptional[List[str]]Top-level section names, in order, used to sort the saved and loaded config. All other sections will be sorted alphabetically.is_interpolatedOptional[bool]Whether the config is interpolated or whether it contains variables. Read from thedataif it’s an instance ofConfigand otherwise defaults toTrue.methodConfig.from_strLoad the config from a string.fromconfectionimportConfigconfig_str="""[training]patience = 10dropout = 0.2"""config=Config().from_str(config_str)print(config["training"])# {'patience': 10, 'dropout': 0.2}}ArgumentTypeDescriptiontextstrThe string config to load.interpolateboolWhether to interpolate variables like${section.key}. Defaults toTrue.overridesDict[str, Any]Overrides for values and sections. Keys are provided in dot notation, e.g."training.dropout"mapped to the value.RETURNSConfigThe loaded config.methodConfig.to_strLoad the config from a string.fromconfectionimportConfigconfig=Config({"training":{"patience":10,"dropout":0.2}})print(config.to_str())# '[training]\npatience = 10\n\ndropout = 0.2'ArgumentTypeDescriptioninterpolateboolWhether to interpolate variables like${section.key}. Defaults toTrue.RETURNSstrThe string config.methodConfig.to_bytesSerialize the config to a byte string.fromconfectionimportConfigconfig=Config({"training":{"patience":10,"dropout":0.2}})config_bytes=config.to_bytes()print(config_bytes)# b'[training]\npatience = 10\n\ndropout = 0.2'ArgumentTypeDescriptioninterpolateboolWhether to interpolate variables like${section.key}. Defaults toTrue.overridesDict[str, Any]Overrides for values and sections. Keys are provided in dot notation, e.g."training.dropout"mapped to the value.RETURNSstrThe serialized config.methodConfig.from_bytesLoad the config from a byte string.fromconfectionimportConfigconfig=Config({"training":{"patience":10,"dropout":0.2}})config_bytes=config.to_bytes()new_config=Config().from_bytes(config_bytes)ArgumentTypeDescriptionbytes_databoolThe data to load.interpolateboolWhether to interpolate variables like${section.key}. Defaults toTrue.RETURNSConfigThe loaded config.methodConfig.to_diskSerialize the config to a file.fromconfectionimportConfigconfig=Config({"training":{"patience":10,"dropout":0.2}})config.to_disk("./config.cfg")ArgumentTypeDescriptionpathUnion[Path, str]The file path.interpolateboolWhether to interpolate variables like${section.key}. Defaults toTrue.methodConfig.from_diskLoad the config from a file.fromconfectionimportConfigconfig=Config({"training":{"patience":10,"dropout":0.2}})config.to_disk("./config.cfg")new_config=Config().from_disk("./config.cfg")ArgumentTypeDescriptionpathUnion[Path, str]The file path.interpolateboolWhether to interpolate variables like${section.key}. Defaults toTrue.overridesDict[str, Any]Overrides for values and sections. Keys are provided in dot notation, e.g."training.dropout"mapped to the value.RETURNSConfigThe loaded config.methodConfig.copyDeep-copy the config.ArgumentTypeDescriptionRETURNSConfigThe copied config.methodConfig.interpolateInterpolate variables like${section.value}or${section.subsection}and
return a copy of the config with interpolated values. Can be used if a config is
loaded withinterpolate=False, e.g. viaConfig.from_str.fromconfectionimportConfigconfig_str="""[hyper_params]dropout = 0.2[training]dropout = ${hyper_params.dropout}"""config=Config().from_str(config_str,interpolate=False)print(config["training"])# {'dropout': '${hyper_params.dropout}'}}config=config.interpolate()print(config["training"])# {'dropout': 0.2}}ArgumentTypeDescriptionRETURNSConfigA copy of the config with interpolated values.methodConfig.mergeDeep-merge two config objects, using the current config as the default. Only
merges sections and dictionaries and not other values like lists. Values that
are provided in the updates are overwritten in the base config, and any new
values or sections are added. If a config value is a variable like${section.key}(e.g. if the config was loaded withinterpolate=False),the
variable is preferred, even if the updates provide a different value. This
ensures that variable references aren’t destroyed by a merge.:warning: Note that blocks that refer to registered functions using the@syntax are only merged if they are referring to the same functions. Otherwise,
merging could easily produce invalid configs, since different functions can
take different arguments. If a block refers to a different function, it’s
overwritten.fromconfectionimportConfigbase_config_str="""[training]patience = 10dropout = 0.2"""update_config_str="""[training]dropout = 0.1max_epochs = 2000"""base_config=Config().from_str(base_config_str)update_config=Config().from_str(update_config_str)merged=Config(base_config).merge(update_config)print(merged["training"])# {'patience': 10, 'dropout': 0.1, 'max_epochs': 2000}ArgumentTypeDescriptionoverridesUnion[Dict[str, Any], Config]The updates to merge into the config.RETURNSConfigA new config instance containing the merged config.Config AttributesArgumentTypeDescriptionis_interpolatedboolWhether the config values have been interpolated. Defaults toTrueand is set toFalseif a config is loaded withinterpolate=False, e.g. usingConfig.from_str.