Package: opset
OpsetA library for simplifying the configuration of Python applications at all stages of deployment.Opset is a config manager that let you manage your configuration via YAML file or environment variables.
The general principle of Opset is that you want to hold your secrets and manage your configurations via
configuration files when doing local development and via environment variables when your app is deployed. It is however
possible to also handle local development through environment variables if the developer sees fit.With Opset you define everything that can be tweaked with your application in
one Pydantic model. This way the developers and integrators working with your code will know exactly what setting they
can change on your code base. You can then overwrite the default config with a local config stored in a file calledlocal.yml, this file is aimed to be used for local development by your developers and let them easily manage a
configuration file that fits their development need. Finally, you can also have environment variables that have a
matching name to your config that will overwrite your config, letting you use your config in a deployed environment
without having your secret written down in a config file. Opset aims to reconcile the ease of use of a
config file with the added security of environment variables.This library is available on PyPI under the name Opset. You can install with pip by runningpip install opset.Table of ContentsLexiconArchitecture OverviewLoading the config for unit testsUsage GuideMaking the difference between null and emptyOpset + Google Cloud Secret ManagerNaming your config sectionsControlling your entry pointsExample Configuration filedefault.ymllocal.ymlunit_test.ymlExample Logging Configuration valuesLog ProcessorsSupport for unit testssetup_unit_test_configmock_configContributing and getting set up for local developmentLexiconTermDefinitionconfigThe config Pydantic model or a configuration file (format: YAML).sectionA section within a configuration file, a section tend to group different settings together under a logical block. For example a section named redis would encompass all settings related specifically to redis.settingA key within a section in a configuration file. A value is associated with a key and querying the config for a setting within a section will return the value associated with it.Architecture OverviewThere are three possible config filesConfig NamePurposeConfig ModelThis is the user defined config model based onOpsetSettingsModel.local.ymlThis is a local config that overwrites the default config, this file is not committed to the repository and is meant to be used in a local development environment.The content of the default config is loaded first, and if any settings are redefined inlocal.yml, the default values
from the model are overwritten bylocal.yml.Environment variables will apply after thelocal.ymloverwrite of the config settings if they have a matching name. To
do so, the environment variable must be named in the following way:{APP_NAME_ALL_CAPS}_{SECTION}_{SETTING}So for the applicationmy-small-projectif we wanted to overwrite the settingportfrom the sectionapp, your
environment variable would need to be named like this:MY_SMALL_PROJECT_APP_PORTIt is also possible to have nested sections, so following the example above, if you wanted to override the value ofapi.weather.hostyou could do so using the following environment variable:MY_SMALL_PROJECT_API_WEATHER_HOSTUsage GuideTo create a new configuration you need to first implement a pydantic model based onOpsetSettingsModelto represent
your configuration. It can contain sub models all based onOpsetSettingsModel. Then you can initialize your
configuration by instantiating theConfigclass. Your config object will be available in theconfigattribute of the
opset config.Opset will also check forlocal.yml. The location needs to be specified when instantiating theConfigobject. Yourlocal.ymlshould be added to.gitignore.A basic Opset setup will look like this:fromopsetimportOpsetSettingsMainModel,ConfigclassMyConfig(OpsetSettingsMainModel):host:strport:int=8080config=Config("my-app",MyConfig,"my_app.config").configYou would then import your newconfigvariable where needed in your app.TheConfigclass takes the following arguments:ParameterDescriptionDefault valueExampleapp_nameThe name of the application, usually the name of the repo. Ex:myproject-example. This will be used for finding the prefix to the environment variables. The name of the app will be uppercased and dashes will be replaced by underscores.myproject-exampleconfig_modelYour implementation ofOpsetSettingsModelthat defines your configuration.config_pathA python path to where the configuration files are. Relative to the application. Ex:tasks.configwould mean that the config files are located in the directory config of the directory tasks from the root of the repo.tasks.configsetup_loggingWhether the logging config should be loaded immediately after the config has been loaded. Your configuration modell will need to have the logging attribute of typeOpsetLoggingConfigfor this to work.  Default toTrue.TrueTrueMaking the difference between null and emptyThe configuration is stored in YAML and follows the YAML standard. As such, it makes a distinction betweennullkeys
and empty keys.app:
  # this setting's value is declared but not defined
  # it will be set to None when accessed unless it is overwritten in local.yml or in an environment variable
  api_key: null
  # this setting's value is set to an empty string
  log_prefix:Controlling your entry pointsThe config object is initiated once you create theConfigobject, before that, trying to get read
a value from the config will throw an exception. It is very important to have a good idea of what the entry points
are in your application and to create yourConfigobject as early as possible in your application to avoid issues.You cannot instantiateConfigmore than one time, so make sure the code handling your configuration is only ran once.Opset + Google Cloud Secret ManagerYou need to install opset with the extrasgcpin order to use this feature.Opset is able to fetch secrets from Google Cloud Secret Manager.
You need to be authenticated usinggcloud CLIor setting up a service
account.The config value should respect on of these formatsopset+gcp://projects/<my_project>/secrets/<my_secret>opset+gcp://projects/<my_project>/secrets/<my_secret>/versions/<my_version>Exampledatabase:host:opset+gcp://projects/dev-3423/secrets/db_hostIt is also possible to create a file.opset.ymlin your project to create mapping for project name.
For instance, with the following config.gcp_project_mapping:dev:dev-3423Opset will be able to map the project name like this.opset+gcp://projects/dev/secrets/db_host -> opset+gcp://projects/dev-3423/secrets/db_hostExample Configuration filelocal.ymlThis file is typically defined by developers for their own development and local usage of the app. This file
may contain secrets and as such it must be added to the.gitignorefile.Example Logging Configuration valuesOpset also provides functionality for configuring the logging handlers for your project, this usesstructlogin the background. This is provided through the aforementionedload_logging_configfunction. If you
choose to use this functionality, you will need to add some more values to your configuration files, and you can find
an example of such values here:logging:date_format:"iso"# strftime-valid date format, e.g.: "%Y-%M-%d", or "iso" to use the standard ISO formatuse_utc:True# Use UTC timezone if true, or local otherwisemin_level:DEBUG# Minimum level to display log forcolors:False# Use colors for log display, defaults to Falsedisable_processors:False# Disables log processors (additional info at the end of the log record)logger_overrides:# overwrite min log level of third party loggersgoogleapiclient:ERRORjson_format:False# Whether the logs should be formatted as json. Defaults to False.Log ProcessorsSince we are usingstructlogyou can use the Processor feature to add additional info to your log records, this
can be useful to add a request ID, or the hostname of the machine to all your log records without having to pass
anything to your logging calls.To use this simply define any processors you want by inheriting from theBaseProcessorclass ofopsetand pass an instance to theload_logging_configon your opset config call:importloggingfromflaskimportFlaskfromopsetimportBaseProcessor,Config,OpsetSettingsMainModel,OpsetLoggingConfig,load_logging_configfrommy_app.request_contextimportget_request_idclassMyConfig(OpsetSettingsMainModel):host:strport:int=8080logging:OpsetLoggingConfigclassRequestContextProcessor(BaseProcessor):def__call__(self,logger,name,event_dict):event_dict["request_id"]=get_request_id()returnevent_dictconfig=Config("my_app",MyConfig,"my_app.config",setup_logging=False).config# Defer the logging setupload_logging_config(config.logging,custom_processors=[RequestContextProcessor()])# Pass your custom processorsapp=Flask(__name__)@app.route("/")defroot():logging.info("This will include the request ID!")return"OK"A processor receives the logger object, the logger name and most importantly theevent_dictwhich contains all the
info of the log record. So simply add to theevent_dictin your processor and return it.In local development processors can add some unnecessary noise to the log output, so they can be disabled by settinglogging.disable_processorstoTruein yourlocal.yml.By default, Opset enables the built-inHostNameProcessor, which adds the machine hostname to log records.
It can be disabled by passinguse_hostname_processor=Falsein theload_logging_configcall.Log HandlersSince we are using python'slogginglibrary, you can use custom log handlers to customize how and where the
information is logged when using the logger.To use this simply define any log handlers you want by inheriting from theHandlerclass ofloggingand overwriting
theemitmethod, and pass an instance to theload_logging_configcall:importloggingfromflaskimportFlaskfromopsetimportConfig,OpsetSettingsMainModel,OpsetLoggingConfig,load_logging_configfromloggingimportHandlerimportjsonclassMyConfig(OpsetSettingsMainModel):host:strport:int=8080logging:OpsetLoggingConfigclassLocalFileHandler(Handler):def__init__(self):Handler.__init__(self)defemit(self,record):"""Will log the record in the root log.json file"""withopen("log.json","w")asfp:json.dump(record.msg,fp)config=Config("my_app",MyConfig,"my_app.config",setup_logging=False).config# Defer the logging setupload_logging_config(config.logging,custom_handlers=[LocalFileHandler()])# Pass your custom handlersapp=Flask(__name__)@app.route("/")defroot():logging.info("Log me in a local file!")return"OK"The handler receives the record object, containing all the log information that was processed by the
processors. The handler can chose what to do with that information, should it be to log it in a local file,
send it to a blob storage, send it to an external tool (ex: Sentry)Support for unit testsOpset support unit testing to make sure you can handle the special cases that may come up in your
application configuration during unit testing.setup_unit_test_configTo setup your config for unit tests you will want to import your opset_config object and callsetup_unit_teston it.
This function takes a dictionary of all the configuration you want to overwrite. This call should happen only once in
your unit test setup.mock_configThemock_configcontextmanager on the opset config is used to temporarily overwrite your configuration. Likesetup_unit_testyou pass a dictionary of the configurations you want to overwrite.Contributing and getting set up for local developmentTo set yourself up for development on Opset, make sure you are usingpoetryand simply run the following commands from the root directory:makeinstall