Configuring the service: Vault+Pydantic
By: Alexander Lavrov
Tags: Python, Config, Kafka, Configuration Management, Vault

Settings classes

DBSetting — database connection

To create the settings class, we must use VaultKVSettings as the base class.

As you can see, config data for the database connection is quite simple. This class will by default look at the secret superapp/dev/db, as we specified in the Config class. At first glance these are simple pydantic.Field, but they all have an extra argument vault_secret_key — it is needed when the key in the secret does not match the name (alias) of the pydantic field in our class, if vault_secret_key is not specified, the provider will search for the key by the field alias.

For example, in our superapp, it is assumed that the superapp/dev/db secret has the “password” and “username” keys, but we want the latter to be placed in the “user” field for convenience and brevity.

Let’s put the following data in the above secret:

{

"host": "testhost",

"password": "testpassword",

"port": "1234",

"username": "testuser"

}

Now, if we run this code, we will get our secret from vault with our DBSettings class:

KafkaSettings — connection to brokers

In this case, let’s imagine that there is one kafka instance for different environments of our service, so the secret is stored along the path superapp/common/kafka:

Note: You can also set secret_path or/and mount_point at the field level so that the provider requests specific values from different secrets (if required). Here is a quote with prioritization of the secret path and mount point from the documentation:

Secret path prioritization:

1. vault_secret_path (Field arg)

2. default_secret_path (Config class field)

3. secret_path (provider initialization optional arg) Mount point prioritization:

1. vault_mount_point (Field arg)

2. default_mount_point (Config class field)

3. mount_point (provider initialization optional arg)

{

"auth_data": "{\"password\": \"testpassword\", \"username\": \"testuser\"}",

"auth_mechanism": "SASL_PLAINTEXT",

"brokers": "kafka://test"

}

Or

{

"auth_data":

{

"password": "testpassword",

"username": "testuser"

},

"brokers": "kafka://test",

"auth_mechanism": "SASL_PLAINTEXT"

}

Note: VaultKVSettings can understand both json and Dict itself.

As a result, this class of settings will be able to collect data from the secret like this:

{

"auth_data":

{

"password": "testpassword",

"username": "testuser"

},

"brokers": "kafka://test",

"auth_mechanism": "SASL_PLAINTEXT"

}

FaustSettings — global configure faust and individual agents

Secret superapp/dev/faust:

{

"agent_concurrency": "5",

"app_name": "superapp-workers",

"partitions_count": "10"

}

If our secret is written as indicated above, then individual agents will take information about the number of partitions in their topics and concurrency from the default fields.

{

"agents": None,

"app_name": "superapp-workers",

"default_concurrency": 5,

"default_partitions_count": 10

}

However, with the help of the AgentConfig model, we can set individual values for a specific agent. For example, if we have agent X with 5 partitions and concurrency 2, then we can change our secret so that information about this agent is in the “agents” field.

Secret superapp/dev/faust:

{

"agent_concurrency": "5",

"agents_specification": {

"X": {

"concurrency": "2",

"partitions": "5"

}

},

"app_name": "superapp-workers",

"partitions_count": "10"

}

If we initialize our settings now, we will receive, in addition to the default fields, also the configuration for a specific agent:

{

"agents":

{

"X":

{

"concurrency": 2,

"partitions": 5

}

},

"app_name": "superapp-workers",

"default_concurrency": 5,

"default_partitions_count": 10

}

Combine the settings classes into a single configuration model

To make our configs even more convenient to use, let’s combine all our settings classes into one model by applying the default_factory:

Now, let’s go to the main.py file and test the collection of complete configuration data for our application:

We get the output of application configuration:

db=DBSettings(user='testuser', password='testpassword', host='testhost', port=1234)



faust=FaustSettings(app_name='superapp-workers', default_partitions_count=10, default_concurrency=5, agents={'X': AgentConfig(partitions=5, concurrency=2)})



kafka=KafkaSettings(auth_mechanism='SASL_PLAINTEXT', brokers='kafka://test', auth_data={'password': 'testpassword', 'username': 'testuser'}) {

"db":

{

"host": "testhost",

"password": "testpassword",

"port": 1234,

"user": "testuser"

},

"faust":

{

"agents":

{

"X":

{

"concurrency": 2,

"partitions": 5

}

},

"app_name": "superapp-workers",

"default_concurrency": 5,

"default_partitions_count": 10

},

"kafka":

{

"auth_data":

{

"password": "testpassword",

"username": "testuser"

},

"brokers": "kafka://test",

"auth_mechanism": "SASL_PLAINTEXT"

}

}

Happiness, joy, delight!