Package: leetconfig
leetconfigIntroductionleetconfigis a package for parsing and aggregating structured configs from multiple
sources, including CLI, YAML, and environment variables. The general workflow is:Define the structure of the config, including detailed information about each option (such as
name, type, help text, etc.). Each option is aConfigEntry, and a group of
these is aConfigGroup.Define an aggregator for the config, including what sources to look at (only parse CLI, parse CLI
and environment variables, parse only YAML, etc.). The aggregator is aConfigParser, and the sources are represented byConfigFormat.At runtime, extract structured config options from the specified sources.ConfigParserTheConfigParseris the top-level object that aggregates all of the options for an application,
along with information about where to find the values for those options. It also provides an entry
point to extract and parse all of the values.ConfigParserExampleLet's begin by saving the following code inleetconfig_readme.py.fromleetconfig.parserimportConfigParsermy_app_parser=ConfigParser("cool_application","a very cool application",sources=[],groups=[],entries=[],)my_app_parser.extract()There are no config options yet, but we already have some functionality. If we run this script with--help, we see:$  python3 leetconfig_readme.py --help
usage: cool application [--help] [--show-config]

a very cool application

Configuration sources (ordered by priority):
  CLI:
    Parse config values from the command line arguments.

Configuration options:This confirms that we have not yet defined any config options. To add config options,ConfigEntrys,ConfigGroups, and/orConfigNamespaces must be defined.ConfigEntryAConfigEntryis the basic unit of this package. EachConfigEntrydefines a single config
option. All config entries must have a parser (aConfigEntryConverterimplementation), a help string, and at least one name. An entry may also have several other flags,
likeis_positionaloris_required. Default values and static lists of choices can also be set
for each entry.Note that if the name of aConfigEntrycontains any underscores_, these are converted to dashes-when parsing from the command line.ConfigEntryConverterConfig options come in a wide variety of formats, so each config entry must define how to
deserialize its options from a raw string. This is done via theConfigEntryConverterimplementation.AConfigEntryConverteris a parser that deserializes an entry's value from string values, and vice
versa. The raw string is parsed from the source (like CLI or YAML) for each config entry and then
fed to theConfigEntryConverterthat is set as theparserfor that respective config entry.Included in this package are many predefined converters such asIntegerConfigEntryConverter,
orEnumConfigEntryConverter. Each expects the raw string to be of a
certain form, and each outputs an instance of the relevant type (intorenum, respectively).Converters can be nested. For example: TheArrayConfigEntryConvertercreates lists of arbitrary
types. How does it know how to parse each element of the list? Well, the constructor for this class
expects to be passed a parser that will be called for each element of the list. In this way one can
parse lists ofints,enums,floats, etc.ConfigEntryandConfigEntryConverterExamplefromleetconfig.parserimportConfigParserfromleetconfig.entryimportConfigEntryfromleetconfig.entry_converterimportStringConfigEntryConverter,IntegerConfigEntryConverterhostname_config=ConfigEntry("hostname",short_names="hn",parser=StringConfigEntryConverter(),help="The host to connect to the server",is_required=True,)# type: ConfigEntry[str]port_config=ConfigEntry("port",short_names="p",parser=IntegerConfigEntryConverter(),help="The port to connect to the server",is_required=True,)# type: ConfigEntry[int]connect_retries_config=ConfigEntry("connect_retries",short_names="cr",parser=IntegerConfigEntryConverter(),help="Max retries when attempting to connect to the server (default 0)",is_required=False,default=0,)# type: ConfigEntry[int]my_app_parser=ConfigParser("cool_application","a very cool application",sources=[],groups=[],entries=[hostname_config,port_config,connect_retries_config],)my_app_parser.extract()hostname=hostname_config.get_value()port=port_config.get_value()connect_retries=connect_retries_config.get_value()print(hostname,port,connect_retries)When the preceding code is saved inleetconfig_readme.pyand then run, it returns the following:$  python3 leetconfig_readme.py --help
usage: cool_application [--help] [--show-config] --hostname VAL --port VAL
                        [--connect-retries VAL]

very cool

Configuration sources (ordered by priority):
  CLI:
    Parse config values from the command line arguments.

Configuration options:
  hostname, hn: string
      The host to connect to the server
  port, p: integer
      The port to connect to the server
  connect_retries, cr: [integer]
      Max retries when attempting to connect to the server (default 0)
        (default: 0)
$  python3 leetconfig_readme.py --hostname localhost --port 1337
localhost 1337 0ConfigGroupAConfigGroupaggregates related config entries together. For example, the options from the
previous example can be bundled into a basic server-clientConfigGroupwith three config entries
namedhostname,port, andconnect-retries:classServerClientConfigGroup(ConfigGroup):def__init__(self):self.hostname=ConfigEntry("hostname",short_names="hn",parser=StringConfigEntryConverter(),help="The host to connect to the server",is_required=True,)# type: ConfigEntry[str]self.port=ConfigEntry("port",short_names="p",parser=IntegerConfigEntryConverter(),help="The port to connect to the server",is_required=True,)# type: ConfigEntry[int]self.connect_retries=ConfigEntry("connect_retries",short_names="cr",parser=IntegerConfigEntryConverter(),help="Max retries when attempting to connect to the server (default 0)",is_required=False,default=0,)# type: ConfigEntry[int]super(ServerClientConfigGroup,self).__init__(entries=[self.hostname,self.port,self.connect_retries],)Groups can be nested. AConfigGroupcan contain otherConfigGroups as well asConfigEntrys.
This allows config groups to compose into a larger config group. For example, we can combine theServerClientConfigGroupwith aPasswordAuthenticationConfigGroupgroup into aServerClientLoginConfigGroupwith five config entries namedhostname,port,connect-retries,username,password:classPasswordAuthenticationConfigGroup(ConfigGroup):def__init__(self):self.username=ConfigEntry("username",short_names="u",parser=StringConfigEntryConverter(),help="The username to authenticate with the service",is_required=True,)# type: ConfigEntry[str]self.password=ConfigEntry("password",short_names="pw",parser=StringConfigEntryConverter(),help="The password to authenticate with the service",is_required=True,)# type: ConfigEntry[str]super(PasswordAuthenticationConfigGroup,self).__init__(entries=[self.username,self.password],)classServerClientLoginConfigGroup(ConfigGroup):def__init__(self):self.server=ServerClientConfigGroup()self.authentication=PasswordAuthenticationConfigGroup()super(ServerClientLoginConfigGroup,self).__init__(groups=[self.server,self.authentication],)ConfigNamespaceAConfigNamespaceis a special case ofConfigGroupand can be used the same ways. The only
difference is that theConfigEntrys are also grouped under a namespace, so that the group can
be distinguished from other groups or entries. This is useful for when there are multiple similar
but distinct configs. For example, one could have multipleServerClientLoginConfigGroupgroups for different services that an
application relies on, and separate them via namespaces:classMyApplicationConfigDefinition(ConfigGroup):def__init__(self):super(MyApplicationConfigDefinition,self).__init__(groups=[ConfigNamespace("redis","r",groups=[ServerClientLoginConfigGroup()]),ConfigNamespace("git","g",groups=[ServerClientLoginConfigGroup()]),],)This group has ten config entries. They are named:redis-hostname,redis-port,redis-connect-retries,redis-username,redis-password,git-hostname,git-port,git-connect-retries,git-username,git-password.Notice that the namespace is prepended to each config entry, making it uniquely identifiable.
Without the ability to create a namespace (a name string), separateConfigEntrieswould have to
be defined for the Redis port and the Git port in order to distinguish those arguments.ConfigNamespaces can thus help reduce duplicated code.We could simplify the example shown above by makingServerClientLoginConfigGroupa namespace from the beginning:classServerClientLoginConfigGroup(ConfigNamespace):def__init__(self,service_name,# type: strservice_short_name,# type: str):self.server=ServerClientConfigGroup()self.authentication=PasswordAuthenticationConfigGroup()super(ServerClientLoginConfigGroup,self).__init__(service_name,service_short_name,groups=[self.server,self.authentication],)classMyApplicationConfigDefinition(ConfigGroup):def__init__(self):super(MyApplicationConfigDefinition,self).__init__(groups=[ServerClientLoginConfigGroup("redis","r"),ServerClientLoginConfigGroup("git","g"),],)This results in the same ten config entries as before.ConfigGroupandConfigNamespaceExampleVery similar implementations of the above definitions ofServerClientConfigGroupandPasswordAuthenticationConfigGroupare already built intoleetconfigasServerClientConfigDefinitionandPasswordAuthenticationConfigDefinition. Using those,
the full previousConfigGroupexample could be written as:fromleetconfig.parserimportConfigParserfromleetconfig.groupimportConfigGroupfromleetconfig.namespaceimportConfigNamespacefromleetconfig.definitions.server_clientimportServerClientConfigDefinitionfromleetconfig.definitions.authenticationimportPasswordAuthenticationConfigDefinitionclassServerClientLoginConfigGroup(ConfigNamespace):def__init__(self,service_name,# type: strservice_short_name,# type: str):self.server=ServerClientConfigDefinition(service_name)self.authentication=PasswordAuthenticationConfigDefinition(service_name,is_required=True)super(ServerClientLoginConfigGroup,self).__init__(service_name,service_short_name,groups=[self.server,self.authentication],)defexport(self):returnself.server.export(),self.authentication.export()classMyApplicationConfigDefinition(ConfigGroup):def__init__(self):self.redis_config=ServerClientLoginConfigGroup("redis","r")self.git_config=ServerClientLoginConfigGroup("git","g")super(MyApplicationConfigDefinition,self).__init__(groups=[self.redis_config,self.git_config],)my_app_config=MyApplicationConfigDefinition()my_app_parser=ConfigParser("cool_application","a very cool application",sources=[],groups=[my_app_config],entries=[],)my_app_parser.extract()redis_options=my_app_config.redis_config.export()git_options=my_app_config.git_config.export()Notice theexportmethod we've added, and thatServerClientConfigDefinitionandPasswordAuthenticationConfigDefinitionalso have this method. This is a common pattern to extract
config options in a convenient form factor, such as a dataclass.Running with--helpwe get:$  python3 leetconfig_readme.py --help
usage: cool_application [--help] [--show-config] --redis-hostname VAL
                        --redis-port VAL [--redis-connect-retries VAL]
                        --redis-username VAL --redis-password VAL
                        --git-hostname VAL --git-port VAL
                        [--git-connect-retries VAL] --git-username VAL
                        --git-password VAL

a very cool application

Configuration sources (ordered by priority):
  CLI:
    Parse config values from the command line arguments.

Configuration options:
  redis:
    hostname, redis_hostname, rhn: string
        The host to connect to the redis server
    port, redis_port, rp: integer
        The port to connect to the redis server
    connect_retries, redis_connect_retries, rcr: [integer]
        Max retries when attempting to connect to the redis server (default 0)
          (default: 0)
    username, redis_username, ru: [string]
        The username to authenticate with the redis service
    password, redis_password, rpw: [string]
        The password to authenticate with the redis service
  git:
    hostname, git_hostname, ghn: string
        The host to connect to the git server
    port, git_port, gp: integer
        The port to connect to the git server
    connect_retries, git_connect_retries, gcr: [integer]
        Max retries when attempting to connect to the git server (default 0)
          (default: 0)
    username, git_username, gu: [string]
        The username to authenticate with the git service
    password, git_password, gpw: [string]
        The password to authenticate with the git serviceSo far, this is a complete and usable config definition. It is annoying to type out all of the flags
every time though, so next we will instead parse these options from a config file.ConfigFormatandsourcesThe role of theConfigFormatis to do the intial extraction of config option values before feeding
each individual option to theConfigEntryConverters to be deserialized. These raw values are
parsed from a source, such CLI arguments. Three mainConfigFormatimplementations exist:CLIConfigFormatRead config options fromsys.argsusing theargparselibrary. This source is always enabled by
default in everyConfigParser. (This can be disabled withforce_no_cliflag toConfigParserconstructor.)YAMLConfigFormatRead config options from one or more YAML config files. The paths to these files are
constructed inone of two ways:TheYAMLConfigFormatis instantiated with one or more file names to find, and optionally with
a list of search directories to check. If no directories are supplied, the current working
directory and/etc/leetconfig/are searched by default. Every file in one of the
search
directories whose name matches a filename will be read. A dictionary of key-value pairs is
aggregated across all opened files and then passed to theConfigEntryConverters for
deserialization.An option--yaml-config-pathis added to the CLI arguments, so a user can specify an absolute
or relative path to a config file. This option is always available in the CLI when aConfigParseris usingYAMLConfigFormat, even ifforce_no_cliis set!EnvironmentConfigFormatReads config options from the OS environment variables.ConfigFormatExampleConsider the following code, saved toleetconfig_readme.py:fromleetconfig.parserimportConfigParserfromleetconfig.groupimportConfigGroupfromleetconfig.namespaceimportConfigNamespacefromleetconfig.definitions.server_clientimportServerClientConfigDefinitionfromleetconfig.definitions.authenticationimportPasswordAuthenticationConfigDefinitionfromleetconfig.format.file_yamlimportYAMLConfigFormatclassServerClientLoginConfigGroup(ConfigNamespace):def__init__(self,service_name,# type: strservice_short_name,# type: str):self.server=ServerClientConfigDefinition(service_name)self.authentication=PasswordAuthenticationConfigDefinition(service_name,is_required=True)super(ServerClientLoginConfigGroup,self).__init__(service_name,service_short_name,groups=[self.server,self.authentication],)defexport(self):returnself.server.export(),self.authentication.export()classMyApplicationConfigDefinition(ConfigGroup):def__init__(self):self.redis_config=ServerClientLoginConfigGroup("redis","r")self.git_config=ServerClientLoginConfigGroup("git","g")super(MyApplicationConfigDefinition,self).__init__(groups=[self.redis_config,self.git_config],)my_app_config=MyApplicationConfigDefinition()my_app_parser=ConfigParser("cool_application","a very cool application",sources=[YAMLConfigFormat("cool_application.config.yaml")],# This is new!groups=[my_app_config],entries=[],)my_app_parser.extract()redis_options=my_app_config.redis_config.export()git_options=my_app_config.git_config.export()When run, notice that there is an additional valid option listed in--help:$  python3 leetconfig_readme.py --help
usage: cool_application [--help] [--show-config] --redis-hostname VAL
                        --redis-port VAL [--redis-connect-retries VAL]
                        --redis-username VAL --redis-password VAL
                        --git-hostname VAL --git-port VAL
                        [--git-connect-retries VAL] --git-username VAL
                        --git-password VAL
                        [--yaml-config-path [VAL [VAL ...]]]

a very cool application

Configuration sources (ordered by priority):
  CLI:
    Parse config values from the command line arguments.
  YAML:
    Parse config values from file(s) named cool_application.config.yaml in the
      current working directory.
[ ... ]A populated a YAML config filecool_application.config.yamlin that path might look like:redis:
  hostname: localhost
  port: 1111
  username: cooluser
  password: "me0w!"
git:
  hostname: localhost
  port: 2222
  username: cooluser
  password: "m3ow?"Alternatively, we could give the file an arbitrary name and pass its path with the--yaml-config-pathoption. We could also define multiple config files, perhaps one for local
and one for remote services. Then, we could pass the local config file when we are testing locally
and, when we are ready to hit the remote services, we can change the entire configuration of the
program by simply passing in a different config file.