Package: pydantic-vault
Pydantic-VaultA simple extension toPydanticBaseSettingsthat can retrieve secrets stored in HashicorpVaultWith Pydantic and Pydantic-Vault, you can easily declare your configuration in a type-hinted class, and load configuration
from environment variables or Vault secrets. Pydantic-Vault will work the same when developing locally (where you probably
login with the Vault CLI and your own user account) and when deploying in production (using a Vault Approle or Kubernetes
authentication for example).InstallationGetting startedDocumentationFieldadditional parametersConfigurationAuthenticationApproleKubernetesVault tokenOrder of priorityLoggingExamplesRetrieve a secret from a KV v2 secret engineRetrieve a whole secret at onceRetrieve a secret from a KV v1 secret engineRetrieve a secret from a database secret engineUse a dynamic path to retrieve secretsKnown limitationsInspirationsLicenseInstallationpipinstallpydantic-vault# or if you use Poetry or Pipenvpoetryaddpydantic-vault
pipenvinstallpydantic-vaultGetting startedWithPydanticBaseSettingsclass, you can easily "create a clearly-defined, type-hinted
application configuration class" that gets its configuration from environment variables. Starting with Pydantic 1.8,custom settings sourcesare officially supported. This is where Pydantic-Vault steps
in, allowing you to load configuration values from Hashicorp Vault secrets. It will work the same when developing locally
(where you probably login with the Vault CLI and your own user account) and when deploying in production (using a Vault
Approle or Kubernetes authentication for example).You can create a normalBaseSettingsclass, and define thecustomise_sources()method to load secrets from your Vault instance using thevault_config_settings_sourcefunction:importosfrompydanticimportBaseSettings,Field,SecretStrfrompydantic_vaultimportvault_config_settings_sourceclassSettings(BaseSettings):# The `vault_secret_path` is the full path (with mount point included) to the secret# The `vault_secret_key` is the specific key to extract from a secretusername:str=Field(...,vault_secret_path="secret/data/path/to/secret",vault_secret_key="my_user")password:SecretStr=Field(...,vault_secret_path="secret/data/path/to/secret",vault_secret_key="my_password",)classConfig:vault_url:str="https://vault.tld"vault_token:SecretStr=os.environ["VAULT_TOKEN"]vault_namespace:str="your/namespace"# Optional, Pydantic-Vault supports Vault namespaces (for Vault Enterprise)@classmethoddefcustomise_sources(cls,init_settings,env_settings,file_secret_settings,):# This is where you can choose which settings sources to use and their priorityreturn(init_settings,env_settings,vault_config_settings_source,file_secret_settings,)settings=Settings()# These variables will come from the Vault secret you configuredsettings.usernamesettings.password.get_secret_value()# Now let's pretend we have already set the USERNAME in an environment variable# (see the Pydantic documentation for more information and to know how to configure it)# With the priority order we defined above, its value will override the Vault secretos.environ["USERNAME"]="my user"settings=Settings()settings.username# "my user", defined in the environment variablesettings.password.get_secret_value()# the value set in VaultDocumentationFieldadditional parametersYou might have noticed that we importFielddirectly from Pydantic. Pydantic-Vault doesn't add any custom logic to it, which means you can still use everything you know and love from Pydantic.The additional parameters Pydantic-Vault uses are:Parameter nameRequiredDescriptionvault_secret_pathYesThe path to your secret in VaultThis needs to be thefull pathto the secret, including its mount point (seeexamplesbelow)vault_secret_keyNoThe key to use in the secretIf it is not specified the whole secret content will be loaded as a dict (seeexamplesbelow)For example, if you create a secretdatabase/prodwith a keypasswordand a value ofa secret passwordin a KV v2 secret engine mounted at the defaultsecret/location, you would access it withpassword:SecretStr=Field(...,vault_secret_path="secret/data/database/prod",vault_secret_key="password")ConfigurationYou can configure the behaviour of Pydantic-Vault in yourSettings.Configclass, or using environment variables:Settings nameTypeRequiredEnvironment variableDescriptioncustomise_sources()YesN/AYou need to implement this function to use Vault as a settings source, and choose the priority order you wantvault_urlstrYesVAULT_ADDRYour Vault URLvault_namespacestr | NoneNoVAULT_NAMESPACEYour Vault namespace (if you use one, requires Vault Enterprise)vault_auth_mount_pointstr | NoneNoVAULT_AUTH_MOUNT_POINTThe mount point of the authentication method, if different from its default mount pointvault_certificate_verifystr | bool | NoneNoVAULT_CA_BUNDLEThe path to a CA bundle validating your Vault certificate, orFalseto disable verification (seehvac docs)Environment variables override what has been defined in theConfigclass.You can also configure everything available in the original PydanticBaseSettingsclass.AuthenticationPydantic-Vault supports the following authentication method (in descending order of priority):direct token authenticationkubernetesapprolePydantic-Vault tries to be transparent and help you work, both during local development and in production. It will try to
find the required information for the first authentication method, if it can't it goes on to the next method, until it
has exhausted all authentication methods. In this case it gives up and logs the failure.You only need to know this order of priority if you specify the authentication parameters for multiple methods.Support is planned for GKE authentication methods (contributions welcome! :wink:).ApproleTo authenticate using theApprole auth method, you need to pass a role ID and a secret ID to your Settings class.Pydantic-Vault reads this information from the following sources (in descending order of priority):theVAULT_ROLE_IDandVAULT_SECRET_IDenvironment variablesthevault_role_idandvault_secret_idconfiguration fields in yourSettings.Configclass (vault_secret_idcan be astror aSecretStr)You can also mix-and-match, e.g. write the role ID in yourSettings.Configclass and retrieve the secret ID from the environment at runtime.Example:frompydanticimportBaseSettings,Field,SecretStrfrompydantic_vaultimportvault_config_settings_sourceclassSettings(BaseSettings):username:str=Field(...,vault_secret_path="path/to/secret",vault_secret_key="my_user")password:SecretStr=Field(...,vault_secret_path="path/to/secret",vault_secret_key="my_password")classConfig:vault_url:str="https://vault.tld"vault_role_id:str="my-role-id"vault_secret_id:SecretStr=SecretStr("my-secret-id")@classmethoddefcustomise_sources(cls,init_settings,env_settings,file_secret_settings,):return(init_settings,env_settings,vault_config_settings_source,file_secret_settings,)KubernetesTo authenticate using theKubernetes auth method, you need to pass a role to your Settings class.Pydantic-Vault reads this information from the following sources (in descending order of priority):theVAULT_KUBERNETES_ROLEenvironment variablethevault_kubernetes_roleconfiguration field in yourSettings.Configclass, which must be astrThe Kubernetes service account token will be read from the file at/var/run/secrets/kubernetes.io/serviceaccount/token.Example:frompydanticimportBaseSettings,Field,SecretStrfrompydantic_vaultimportvault_config_settings_sourceclassSettings(BaseSettings):username:str=Field(...,vault_secret_path="path/to/secret",vault_secret_key="my_user")password:SecretStr=Field(...,vault_secret_path="path/to/secret",vault_secret_key="my_password")classConfig:vault_url:str="https://vault.tld"vault_kubernetes_role:str="my-role"@classmethoddefcustomise_sources(cls,init_settings,env_settings,file_secret_settings,):return(init_settings,env_settings,vault_config_settings_source,file_secret_settings,)Vault tokenTo authenticate using theToken auth method, you need to pass a Vault token to yourSettingsclass.Pydantic-Vault reads this token from the following sources (in descending order of priority):theVAULT_TOKENenvironment variablethe~/.vault-tokenfile (so you can use thevaultCLI to login locally, Pydantic-Vault will transparently reuse its token)thevault_tokenconfiguration field in yourSettings.Configclass, which can be astror aSecretStrExample:frompydanticimportBaseSettings,Field,SecretStrfrompydantic_vaultimportvault_config_settings_sourceclassSettings(BaseSettings):username:str=Field(...,vault_secret_path="path/to/secret",vault_secret_key="my_user")password:SecretStr=Field(...,vault_secret_path="path/to/secret",vault_secret_key="my_password")classConfig:vault_url:str="https://vault.tld"vault_token:SecretStr=SecretStr("my-secret-token")@classmethoddefcustomise_sources(cls,init_settings,env_settings,file_secret_settings,):return(init_settings,env_settings,vault_config_settings_source,file_secret_settings,)Order of priorityThanks to the new feature in Pydantic 1.8 that allows you tocustomize settings sources, you can choose the order of priority you want.Here are some examples:frompydanticimportBaseSettingsfrompydantic_vaultimportvault_config_settings_sourceclassSettings(BaseSettings):"""In descending order of priority:- arguments passed to the `Settings` class initializer- environment variables- Vault variables- variables loaded from the secrets directory, such as Docker Secrets- the default field values for the `Settings` model"""classConfig:@classmethoddefcustomise_sources(cls,init_settings,env_settings,file_secret_settings,):return(init_settings,env_settings,vault_config_settings_source,file_secret_settings,)classSettings(BaseSettings):"""In descending order of priority:- Vault variables- environment variables- variables loaded from the secrets directory, such as Docker Secrets- the default field values for the `Settings` modelHere we chose to remove the "init arguments" source,and move the Vault source up before the environment source"""classConfig:@classmethoddefcustomise_sources(cls,init_settings,env_settings,file_secret_settings,):return(vault_config_settings_source,env_settings,file_secret_settings)LoggingThe library exports a logger calledpydantic-vault.To help debugging you can change the log level. A simple way to do that if you do not have a custom log setup is:# At the beginning of your main file or entrypointimportlogginglogging.basicConfig()logging.getLogger("pydantic-vault").setLevel(logging.DEBUG)# Change the log level hereExamplesAll examples use the following structure, so we will omit the imports and theConfiginner class:frompydanticimportBaseSettings,Field,SecretStrfrompydantic_vaultimportvault_config_settings_sourceclassSettings(BaseSettings):################################################ THIS PART CHANGES IN THE DIFFERENT EXAMPLES #username:str=Field(...,vault_secret_path="secret/data/path/to/secret",vault_secret_key="my_user")###############################################classConfig:vault_url:str="https://vault.tld"@classmethoddefcustomise_sources(cls,init_settings,env_settings,file_secret_settings,):return(init_settings,env_settings,vault_config_settings_source,file_secret_settings,)Retrieve a secret from a KV v2 secret engineSuppose your secret is atmy-api/prodand looks like this:Key             Value
---             -----
root_user       root
root_password   a_v3ry_s3cur3_p4ssw0rdYour settings class would be:classSettings(BaseSettings):# The `vault_secret_path` is the full path (with mount point included) to the secret.# For a KV v2 secret engine, there is always a `data/` sub-path between the mount point and# the secret actual path, eg. if your mount point is `secret/` (the default) and your secret# path is `my-api/prod`, the full path to use is `secret/data/my-api/prod`.# The `vault_secret_key` is the specific key to extract from a secret.username:str=Field(...,vault_secret_path="secret/data/my-api/prod",vault_secret_key="root_user")password:SecretStr=Field(...,vault_secret_path="secret/data/my-api/prod",vault_secret_key="root_password",)settings=Settings()settings.username# "root"settings.password.get_secret_value()# "a_v3ry_s3cur3_p4ssw0rd"Retrieve a whole secret at onceIf you omit thevault_secret_keyparameter in yourField, Pydantic-Vault will load
the whole secret in your class field.With the same secret as before, located atmy-api/prodand with this data:Key             Value
---             -----
root_user       root
root_password   a_v3ry_s3cur3_p4ssw0rdYou could use a settings class like this to retrieve everything in the secret:classSettings(BaseSettings):# The `vault_secret_path` is the full path (with mount point included) to the secret.# For a KV v2 secret engine, there is always a `data/` sub-path between the mount point and# the secret actual path, eg. if your mount point is `secret/` (the default) and your secret# path is `my-api/prod`, the full path to use is `secret/data/my-api/prod`.# We don't pass a `vault_secret_key` here so that Pydantic-Vault fetches all fields at once.credentials:dict=Field(...,vault_secret_path="secret/data/my-api/prod")settings=Settings()settings.credentials# { "root_user": "root", "root_password": "a_v3ry_s3cur3_p4ssw0rd" }You can also use a PydanticBaseModelclass to parse and validate the incoming secret:classCredentials(BaseModel):root_user:strroot_password:SecretStrclassSettings(BaseSettings):# The `vault_secret_path` is the full path (with mount point included) to the secret.# For a KV v2 secret engine, there is always a `data/` sub-path between the mount point and# the secret actual path, eg. if your mount point is `secret/` (the default) and your secret# path is `my-api/prod`, the full path to use is `secret/data/my-api/prod`.# We don't pass a `vault_secret_key` here so that Pydantic-Vault fetches all fields at once.credentials:Credentials=Field(...,vault_secret_path="secret/data/my-api/prod")settings=Settings()settings.credentials.root_user# "root"settings.credentials.root_password.get_secret_value()# "a_v3ry_s3cur3_p4ssw0rd"Retrieve a secret from a KV v1 secret engineSuppose your secret is atmy-api/prodand looks like this:Key             Value
---             -----
root_user       root
root_password   a_v3ry_s3cur3_p4ssw0rdYour settings class would be:classSettings(BaseSettings):# The `vault_secret_path` is the full path (with mount point included) to the secret.# For a KV v1 secret engine, the secret path is directly appended to the mount point,# eg. if your mount point is `kv/` (the default) and your secret path is `my-api/prod`,# the full path to use is `kv/my-api/prod` (unlike with KV v2 secret engines).# The `vault_secret_key` is the specific key to extract from a secret.username:str=Field(...,vault_secret_path="kv/my-api/prod",vault_secret_key="root_user")password:SecretStr=Field(...,vault_secret_path="kv/my-api/prod",vault_secret_key="root_password")settings=Settings()settings.username# "root"settings.password.get_secret_value()# "a_v3ry_s3cur3_p4ssw0rd"⚠ Beware of theknown limitationson KV v1 secrets!Retrieve a secret from a database secret engineDatabase secrets can be "dynamic", generated by Vault every time you request access.
Because every call to Vault will create a new database account, you cannot store the username
and password in two different fields in your settings class, or you would get the username of thefirstgenerated account and the password of thesecondaccount. This means that you mustnotpass avault_secret_key, so that Pydantic-Vault retrieves the whole secret at once.You can store the credentials in a dict or in a customBaseModelclass:classDbCredentials(BaseModel):username:strpassword:SecretStrclassSettings(BaseSettings):# The `vault_secret_path` is the full path (with mount point included) to the secret.# For a database secret engine, the secret path is `<mount point>/creds/<role name>`.# For example if your mount point is `database/` (the default) and your role name is# `my-db-prod`, the full path to use is `database/creds/my-db-prod`. You will receive# `username` and `password` fields in response.# You must *not* pass a `vault_secret_key` so that Pydantic-Vault fetches both fields at once.db_creds:DbCredentials=Field(...,vault_secret_path="database/creds/my-db-prod")db_creds_in_dict:dict=Field(...,vault_secret_path="database/creds/my-db-prod")settings=Settings()settings.db_creds.username# "generated-username-1"settings.db_creds.password.get_secret_value()# "generated-password-for-username-1"settings.db_creds_in_dict["username"]# "generated-username-2"settings.db_creds_in_dict["password"]# "generated-password-for-username-2"Use a dynamic path to retrieve secretsIf you have different paths for your secrets (for example if you have different environments) you can use string formatting
to dynamically generate the paths depending on an environment variable.importos# You will need to specify the environment in an environment variable, but by# default it falls back to "dev"ENV=os.getenv("ENV","dev")classSettings(BaseSettings):# This will load different secrets depending on the value of the ENV environment variableusername:str=Field(...,vault_secret_path=f"kv/my-api/{ENV}",vault_secret_key="root_user")password:SecretStr=Field(...,vault_secret_path=f"kv/my-api/{ENV}",vault_secret_key="root_password")settings=Settings()settings.username# "root"settings.password.get_secret_value()# "a_v3ry_s3cur3_p4ssw0rd"Known limitationsOn KV v1 secret engines, if your secret has adatakey and you do not specify avault_secret_keyto load the whole secret at once, Pydantic-Vault will only load the content of thedatakey.
For example, with a secretkv/my-secretKey             Value
---             -----
user            root
password        a_v3ry_s3cur3_p4ssw0rd
data            a very important piece of dataand the settings classclassSettings(BaseSettings):my_secret:dict=Field(...,vault_secret_path="kv/my-secret")Pydantic-Vault will try to load only thedatavalue (a very important piece of data) inmy_secret, which will fail validation from Pydantic because it is not a dict.Workaround:Rename thedatakey in your secret 😅Workaround:Migrate to KV v2On Pydantic version before 1.10.3, Pydantic by default takes up ~50-80 MB, because it is compiled to a native extension
and optimized for speed instead of file size. If you don't rely much on Pydantic (you only use it for your app configuration
with Pydantic-Vault, you parse/serialize a low volume of JSON, your code is generally slow and Pydantic wouldn't be the
bottleneck) you can use the flag--no-binary pydanticwhen runningpip installto install the pure-Python version
instead of the compiled one (which comes at less than 1 MB). You can also add the flag on its own line in yourrequirements.txt. See this discussionhttps://github.com/samuelcolvin/pydantic/issues/2276for more information.This has been fixed in Pydantic 1.10.3, and a default install with a precompiled wheel now takes ~8 MB.InspirationsAnsiblehashi_vaultlookup pluginfor the API and some codeHashicorp's Vault GitHub Actionfor the APILicensePydantic-Vault is available under theMIT license.