Repository: freqtrade/freqtrade

Issue:
Discover configuration/secrets via environment variables
<!-- 
Note: this section will not show up in the issue.
Have you search for this feature before requesting it? It's highly likely that a similar request was already filed.
-->

## Describe the enhancement

I'd like to be able to inject configuration (specially secrets like tokens, credentials, and keys) via environment variables. This would make it easier to deploy freqtrade using declarative tools like Terraform, Ansible, Chef, etc., or on workload orchestration platforms like Kubernetes or Nomad, while avoiding storing sensitive secrets in source control.

Ideally, I 'd like to be able to do something along the lines of:
```
export FREQTRADE_CONFIG_EXCHANGE_SECRET=my-exchange-secret
export FREQTRADE_CONFIG_TELEGRAM_TOKEN=my-telegram-token
export FREQTRADE_CONFIG_API_SERVER_PASSWORD=my-api-server-password
freqtrade trade --config user_data/config.json ...
```
And both the values in `config.json` would be merged with the secrets injected via `FREQTRADE_CONFIG_*` environment variables. This way we can separate the workload invocation from the configuration, making it more flexible to deploy.

Other non-sensitive configuration values doesn't have to be injectable via this mechanism, but it may make sense to allow anything for consistency, possibly with some nomenclature that naturally maps to the configuration schema, e.g. something like:
```
"ask_strategy": {
  "ignore_buying_expired_candle_after": 300,
  "price_side": "bid"
}
```
Can be injected as:
```
FREQTRADE_CONFIG__ASK_STRATEGY__IGNORE_BUYING_EXPIRED_CANDLE_AFTER=300
FREQTRADE_CONFIG__ASK_STRATEGY__PRICE_SIDE=bid
```
In other words, use `FREQTRADE_CONFIG` as a prefix, and `__` as a separator to indicate hierarchical levels of the configuration.

I hope this explains the feature request!


Discussion:
freqtrade supports multiple configuration files - so you can use `--config commonconfig.json --config sensitivevalues.json` which will then be merged, so you can avoid storing sensitive information in different, non-sourcecontrolled files.

You can use .gitignore for the sensitivevalues.json file (or some other name you like to use) to avoid storing these in source control.

because of that, i don't think that supporting environment variables makes sense, especially considering that our config logic is already quite complex (some values can come from either command line, configuration or strategy) - adding a 4th option to this will not make sense.

it'll also not improve security, as in both config or environment variable case, you'll have to store the variable in a file to allow the bot to start (the only difference is if it's a .env file, or a json file).
The full config will also be in memory all the time, as the sensitive data is needed by the program to operate, so the environment variables will also not give you any operational security either.
In the context of a distributed, cloud-native application, relying on `.gitignore` is usually a non-starter, since it's meant to be used for a local development workflow, but not for CI/CD pipelines, container orchestration systems, or virtualization platforms, where `.gitignore`'d files aren't persisted across time or system boundaries.

Storing plain-text secrets in memory is ok from a security point of view, because usually, for an attacker to have access to the operating system's memory, it means many other barriers were broken first, so a secret in memory is far from the weakest link. Instead, the issue becomes how is that secret securely bootstrapped into an immutable environment, while minimizing the time it exists in plain text (including on files in a local developer machine that are being excluded by `.gitifnore`), and that's how security is improved. There are widely-used solutions and patterns in the cloud-native space (specifically in the Kubernetes ecosystem), that accomplish this in a way that's completely application-agnostic and future-proof. Lastly, storing configuration via environment variables is one of the "factors" or a best practice in [Twelve-Factor Apps](https://12factor.net/config), which is a popular software development methodology.

In any case, I understand if you don't consider this a viable feature of freqtrade, and I appreciate you taking the time to respond. I just wanted to provide more perspective into the reason why I think this is a desirable feature from the point of view of running freqtrade on a distributed system.

Thanks!
If you think it's a must, feel free to contribute it that way in a clean PR (including tests, with docs).

we've had another attempt for this in #4409 recently, but allowing the user to pick his environment variables is asking for troubles, as then it'll become confusing for non-experienced users.

Personally, i don't use kubernetes for freqtrade, but plain docker (with compose).

Also, i don't see any benefit on running it on a kubernetes cluster, as the main benefit is the resiliance and scaling - but the moment you run 2 docker instances with the same config, telegram will say "no" - and storage for the database will be an issue too.

Also, all cloud instances of kubernetes i've seen so far cost way more than a simple VPS with docker installed (simply because you'll need multiple even if you run only one bot) - but i guess that's me trying to save money - and not the general consens (where cloud seems to be allowed to cost whatever it costs).
I opened the issue more as a way to track the potential feature than strictly requesting it. I'm not the most experienced Python developer, but if there is enough support or interest in this, I might take a stab at it. In the meantime, I defer to you or other contributors to decide.

Workload redundancy isn't the only benefit of Kubernetes, but rather it's the self-healing, i.e. the ability to automatically restart the workload in case of unexpected errors, replace the workload if the node crashes, as well as other features like reserve CPU/memory resources, monitoring/alerting, etc. It can also make it easier to run hyperopt in a distributed/parallel way, since you don't have to decide where to run what, you just submit the workload and Kubernetes does the orchestration for you. As far as storage, there are already a handful of solutions for managing distributed storage, so that wouldn't be an issue either.

Certainly, a production-grade Kubernetes control plane isn't usually justified for a single service, because there are financial costs as you point out, but also intangible maintenance costs over the long run. In my personal case, I have a home Kubernetes cluster where I'm already running many other services so I benefit from the binpacking, and I don't have to pay a cloud provider for either the control plane or the compute. Having a Kubernetes cluster at home is probably not the norm, but my point is that in general, whether cloud or on-prem, the cost of adding the `N+1`th service to an existing cluster nears 0 as the value of `N` increases. Some services like GKE (if used minimally) typically fall below their [free tier](https://cloud.google.com/kubernetes-engine/pricing#:~:text=The%20cluster%20management%20fee%20of,to%20zonal%20and%20Autopilot%20clusters.), so the financial burden can be less of a concern.

Lastly, Kubernetes isn't limited to cloud or remote instances, there are projects like [k3d](https://k3d.io/), [minikube](https://minikube.sigs.k8s.io/docs/), or [kind](https://kind.sigs.k8s.io/) that help you run Kubernetes locally.

I'll gladly continue discussing the benefits of cloud-native infrastructure or answer any additional questions people might have. For now, I'd be happy with just leaving this issue open and see if it gets some traction, and figure out if there's a follow-up action item. Thanks!
i don't think this is about kubernetes, but about environment variables in general.

If it were about kubernetes, you could use secrets (which can be mounted as files iirc) to store the configuration.
Depending on the controlpane (i use rancher at work) - you can make the changes in a webui.

---

Returning to the topic - i'm not a huge fan of environment variables, as they'll add a 4th way to change configuration values (we currently have config(s) -> strategy -> defaults) - and adding environment variables at the front has the potential to confuse people.

If any, i'd use the ["airflow" variant](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html?highlight=environment#environment-variables) (they most likely didn't invent it - but i came across this way with them first), which would have a variable as `FREQTRADE__{section}__{key}`  (e.g. `FREQTRADE__ASK_STRATEGY__PRICE_SIDE = 'bid'`).

I find that's a clean (and pretty flexible) approach to defining variables in a way that makes it clear where it'll be loaded (obviously i'd want that in the startup log too - so it's very much clear where it came from).
> I opened the issue more as a way to track the potential feature than strictly requesting it. I'm not the most experienced Python developer, but if there is enough support or interest in this, I might take a stab at it. In the meantime, I defer to you or other contributors to decide.
> 
> Workload redundancy isn't the only benefit of Kubernetes, but rather it's the self-healing, i.e. the ability to automatically restart the workload in case of unexpected errors, replace the workload if the node crashes, as well as other features like reserve CPU/memory resources, monitoring/alerting, etc. It can also make it easier to run hyperopt in a distributed/parallel way, since you don't have to decide where to run what, you just submit the workload and Kubernetes does the orchestration for you. As far as storage, there are already a handful of solutions for managing distributed storage, so that wouldn't be an issue either.
> 
> Certainly, a production-grade Kubernetes control plane isn't usually justified for a single service, because there are financial costs as you point out, but also intangible maintenance costs over the long run. In my personal case, I have a home Kubernetes cluster where I'm already running many other services so I benefit from the binpacking, and I don't have to pay a cloud provider for either the control plane or the compute. Having a Kubernetes cluster at home is probably not the norm, but my point is that in general, whether cloud or on-prem, the cost of adding the `N+1`th service to an existing cluster nears 0 as the value of `N` increases. Some services like GKE (if used minimally) typically fall below their [free tier](https://cloud.google.com/kubernetes-engine/pricing#:~:text=The%20cluster%20management%20fee%20of,to%20zonal%20and%20Autopilot%20clusters.), so the financial burden can be less of a concern.
> 
> Lastly, Kubernetes isn't limited to cloud or remote instances, there are projects like [k3d](https://k3d.io/), [minikube](https://minikube.sigs.k8s.io/docs/), or [kind](https://kind.sigs.k8s.io/) that help you run Kubernetes locally.
> 
> I'll gladly continue discussing the benefits of cloud-native infrastructure or answer any additional questions people might have. For now, I'd be happy with just leaving this issue open and see if it gets some traction, and figure out if there's a follow-up action item. Thanks!

I'm really interested if you found a good way to deploy fretrade on a home kubernetes cluster. I want to give it a stab with raspi and k3s or similar this weekend, so any thing to get this of the floor would be appreciated.

Currently I only have one raspi, so I'm mainly interested in the self-healing (and maybe easier management) of running multiple strategies on a single node. As you described, scaling over multiple nodes is not important to me...
@JoeSchr I created a very rough version of a [Helm chart for freqtrade](https://github.com/patoarvizu/freqtrade-helm-chart) that might help you get started. Issues and PRs are definitely welcome!

Resolving patch:
diff --git a/freqtrade/configuration/configuration.py b/freqtrade/configuration/configuration.py
--- a/freqtrade/configuration/configuration.py
+++ b/freqtrade/configuration/configuration.py
@@ -11,6 +11,7 @@
 from freqtrade.configuration.check_exchange import check_exchange
 from freqtrade.configuration.deprecated_settings import process_temporary_deprecated_settings
 from freqtrade.configuration.directory_operations import create_datadir, create_userdata_dir
+from freqtrade.configuration.environment_vars import enironment_vars_to_dict
 from freqtrade.configuration.load_config import load_config_file, load_file
 from freqtrade.enums import NON_UTIL_MODES, TRADING_MODES, RunMode
 from freqtrade.exceptions import OperationalException
@@ -71,6 +72,11 @@ def load_from_files(self, files: List[str]) -> Dict[str, Any]:
 
             # Merge config options, overwriting old values
             config = deep_merge_dicts(load_config_file(path), config)
+
+        # Load environment variables
+        env_data = enironment_vars_to_dict()
+        config = deep_merge_dicts(env_data, config)
+
         config['config_files'] = files
         # Normalize config
         if 'internals' not in config:
diff --git a/freqtrade/configuration/environment_vars.py b/freqtrade/configuration/environment_vars.py
new file mode 100644
--- /dev/null
+++ b/freqtrade/configuration/environment_vars.py
@@ -0,0 +1,54 @@
+import logging
+import os
+from typing import Any, Dict
+
+from freqtrade.constants import ENV_VAR_PREFIX
+from freqtrade.misc import deep_merge_dicts
+
+
+logger = logging.getLogger(__name__)
+
+
+def get_var_typed(val):
+    try:
+        return int(val)
+    except ValueError:
+        try:
+            return float(val)
+        except ValueError:
+            if val.lower() in ('t', 'true'):
+                return True
+            elif val.lower() in ('f', 'false'):
+                return False
+    # keep as string
+    return val
+
+
+def flat_vars_to_nested_dict(env_dict: Dict[str, Any], prefix: str) -> Dict[str, Any]:
+    """
+    Environment variables must be prefixed with FREQTRADE.
+    FREQTRADE__{section}__{key}
+    :param env_dict: Dictionary to validate - usually os.environ
+    :param prefix: Prefix to consider (usually FREQTRADE__)
+    :return: Nested dict based on available and relevant variables.
+    """
+    relevant_vars: Dict[str, Any] = {}
+
+    for env_var, val in sorted(env_dict.items()):
+        if env_var.startswith(prefix):
+            logger.info(f"Loading variable '{env_var}'")
+            key = env_var.replace(prefix, '')
+            for k in reversed(key.split('__')):
+                val = {k.lower(): get_var_typed(val) if type(val) != dict else val}
+            relevant_vars = deep_merge_dicts(val, relevant_vars)
+
+    return relevant_vars
+
+
+def enironment_vars_to_dict() -> Dict[str, Any]:
+    """
+    Read environment variables and return a nested dict for relevant variables
+    Relevant variables must follow the FREQTRADE__{section}__{key} pattern
+    :return: Nested dict based on available and relevant variables.
+    """
+    return flat_vars_to_nested_dict(os.environ.copy(), ENV_VAR_PREFIX)
diff --git a/freqtrade/constants.py b/freqtrade/constants.py
--- a/freqtrade/constants.py
+++ b/freqtrade/constants.py
@@ -47,6 +47,7 @@
 USERPATH_NOTEBOOKS = 'notebooks'
 
 TELEGRAM_SETTING_OPTIONS = ['on', 'off', 'silent']
+ENV_VAR_PREFIX = 'FREQTRADE__'
 
 
 # Define decimals per coin for outputs


Test patch:
diff --git a/tests/test_configuration.py b/tests/test_configuration.py
--- a/tests/test_configuration.py
+++ b/tests/test_configuration.py
@@ -18,8 +18,9 @@
                                                          process_deprecated_setting,
                                                          process_removed_setting,
                                                          process_temporary_deprecated_settings)
+from freqtrade.configuration.environment_vars import flat_vars_to_nested_dict
 from freqtrade.configuration.load_config import load_config_file, load_file, log_config_error_range
-from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL
+from freqtrade.constants import DEFAULT_DB_DRYRUN_URL, DEFAULT_DB_PROD_URL, ENV_VAR_PREFIX
 from freqtrade.enums import RunMode
 from freqtrade.exceptions import OperationalException
 from freqtrade.loggers import _set_loggers, setup_logging, setup_logging_pre
@@ -1349,3 +1350,35 @@ def test_process_deprecated_ticker_interval(mocker, default_conf, caplog):
     with pytest.raises(OperationalException,
                        match=r"Both 'timeframe' and 'ticker_interval' detected."):
         process_temporary_deprecated_settings(config)
+
+
+def test_flat_vars_to_nested_dict(caplog):
+
+    test_args = {
+        'FREQTRADE__EXCHANGE__SOME_SETTING': 'true',
+        'FREQTRADE__EXCHANGE__SOME_FALSE_SETTING': 'false',
+        'FREQTRADE__EXCHANGE__CONFIG__whatever': 'sometime',
+        'FREQTRADE__ASK_STRATEGY__PRICE_SIDE': 'bid',
+        'FREQTRADE__ASK_STRATEGY__cccc': '500',
+        'FREQTRADE__STAKE_AMOUNT': '200.05',
+        'NOT_RELEVANT': '200.0',  # Will be ignored
+    }
+    expected = {
+        'stake_amount': 200.05,
+        'ask_strategy': {
+            'price_side': 'bid',
+            'cccc': 500,
+        },
+        'exchange': {
+            'config': {
+                'whatever': 'sometime',
+            },
+            'some_setting': True,
+            'some_false_setting': False,
+        }
+    }
+    res = flat_vars_to_nested_dict(test_args, ENV_VAR_PREFIX)
+    assert res == expected
+
+    assert log_has("Loading variable 'FREQTRADE__EXCHANGE__SOME_SETTING'", caplog)
+    assert not log_has("Loading variable 'NOT_RELEVANT'", caplog)