Project: Vector (Observability and Analysis / Observability)
File: 2022-02-24-11552-dd-agent-style-secret-management.md

The Datadog Agent has a straightforward secret resolution facility to avoid having sensitive information stored directly in its config, it relies on a user-provided external program that is run to retrieve sensitive value from a third party system. This RFC aims to propose a similar mechanism for Vector. The provide all user-relevant information to use that feature. It covers the specification for the user-provided executable that loads/decrypts secrets along with the agent configuration and the syntax to retrieve encrypted config value. The ongoing . `vector config` ()and related Vector enterprise work. User shall be able to use the same kind of executable to load/decrypt secrets for Vector. This new feature will have a deterministic behaviour when used in conjunction with templates. Situations like topology reload, including SIGHUP configuration reload where secret shall also be refreshed, Load/decryption failure shall be accounted for. Integration with other secret or distribution configuration like vault, however this RFC will account for that kind of future extension As of today, secrets like authentication tokens and passwords should be provided inside the topology configuration in plain text or in and that may not be acceptable in some circumstances. Decoupling secret management and key rotation from configuration management. Use the same kind of API between Vector and a user-provided executable as the [one between the Agent and the secret retrieving executable][dd-secret-backend-exec-api]. A set of top level options like the ones the Datadog Agent . The user will use the following placeholder : `SECRET[<backend>.<secret_key>]` to indicate Vector that a secret should be retrieved for this option (the Datadog Agent uses `ENC[secret_key]`, ENC stands for encrypted, but that can be confusing, hence the current suggestion to use `SECRET[<backend>.<secret_key>]` instead). Datadog Secret API, as per the official : Communication happens on stdout/stdin It uses plain text json Vector would run the user-provided executable and feed its standard input with the list of secrets to retrieve: ```json {"version": "1.0", "secrets": ["secret1", "secret2"]} ``` The version used by the Datadog agent is 1.0. The user provided executable should then reply on its standard output with a JSON formatted string: ```json { "secret1": {"value": "secret_value", "error": null}, "secret2": {"value": null, "error": "could not fetch the secret"} } ``` Note: The version field can be used to introduce specific behaviour, one major useful thing that could be introduced is the ability for the user provided executable to provide a secrete expiration date. This is mentioned in the improvements section. In the placeholder syntax `SECRET[<backend>.<secret_key>]`, dots are not allowed in `<backend>` but can be used in `<secret_key>`, this means that `SECRET[mul.ti.ple.dots]` will cause Vector to query the backend named `mul` (it shall be defined) for the secret key `ti.ple.dots` New top level options to be added will be sitting inside the `secret` namespace, this would lead to something like: ```toml [secret.local_exec] type = "exec" path = "/path/to/the/command" argument = "--config foo=bar" timeout = 5 [secret.prod_vault] type = "vault" address = "https://vault.corp.tld/" token = "SECRET[localexec.vaulttoken]" timeout = 5 [sources.system_logs] type = "file" includes = ["/var/log/system.log"] [sinks.app_logs] type = "datadog_logs" defaultapikey = "SECRET[prodvault.ddapi202202]" inputs = ["system_logs"] ``` The first implementation would only support the `exec` backend but with extensibility point clearly identified to easily implement additional backend if needed. Overall the behaviour for corner cases should follow what's in place for environment variable interpolation as this is a very close feature. When a configuration reload happens, secrets shall also be refreshed. A secret backend that would lie in

and would call the user provided executable for secrets, cache secrets to avoid calling the backend for the same key multiple time. It would read the configuration file once to get its config before further processing, ideally env var interpolating should be supported (this should not be a problem). In `./src/config/builder.rs`, `loadbuilderfrom_paths` will still be returning a complete configuration with placeholders replaced by secrets. The `ConfigBuilder` struct will get a new `secret` field (typed to something like `IndexMap<ComponentKey,SecretBackend>`), this means that `loadbuilderfrom_paths` will then assert if this field is not empty before returning to the caller, and if it, the config will be reloaded with this `SecretBackend` passed to downstream callee and hook the secret interpolation around the same point as [the environment variable interpolation][env-var-hook]: All `SECRET[<backend>.<key>]` placeholders present in config will be collected. Every backend that is specified in this secret list will be queried for all the secrets it should be used for retrieval. And then the interpolation will actually happen (only if all secret were retrieved successfully). Note: As shown in the aforementioned config sample a secret backend shall itself be able to use other secret backend for its own initialisation. The implementation should ease future extension and split the internal API queried by the interpolation logic and the secret provider that may see other implementation like: `exec` (the one we will focus on first), `vault`, `k8s-config-map`, `aws-secretsmanager`, etc. Some users just can't put sensitive information inside their configuration. Using environment variable violates security requirements for sensitive environments since environment variables can be leaked by an attacker and access to those variables is complex to audit. Integrating with other third party tools directly like Vault would provide better error management and avoid relying on a custom, user-provided binary. This binary might have to be injected into container images which may be inconvenient, other options like using an external volume may be more acceptable but it would still involve a third party executable and all the associated risks. The Datadog Agent has exactly the same feature, despite its simple approach it works reasonably well, but it is cannot easily support advanced secret management like certificate distribution/revocation, key rotation, etc. Vault is the standard in the industry, and it comes with all kind of advanced features that cannot really supported by the user-provided executable solution. Integrate with other third party tools: Vault and CSP APIs for secret management to start with. Stick to environment variables interpolation and leverage as environment variables, relevant examples are already in the . Note that the Datadog Agent is now capable to . For the secret syntax in config other possible placeholder could have been used: the Datadog Agent uses `ENC[secret_key]`, and another possible URI-like solution was mentioned (`secret://<backend>/<key>`). Note: doing nothing is not really an alternative here, as plain text secret in config is a strong blocker for some users. Sticking to env var from K8s secret still seems a reasonable approach as K8s is the reference deployment in many situations. Specific security constraints that may have been missed (enforcing the same set of constraints the Datadog Agent uses sound like a reasonable approach to start with, / ). [ ] Implement the secret backend logic with the minimal set of options on all supported platform. [ ] Allow a secret backend config to leverage another secret backend for its own config. [ ] Document typical usecases. Support additional backend. Embed/implement helpers like the . Possible extension to the API, it will mostly depends on user feedback