Repository: QuantConnect/lean-cli (python)

Issue:
DataProvider not valid for `_configure_environment()` in Lean v1.0.178
When you have tried to run backtest with **any** `--data-provider` I got error below.
### Error Message
```
D:\source>lean backtest --data-provider Polygon IQFeedAlgo --verbose
Context information:
  Python version: 3.8.10 (tags/v3.8.10:3d8993a, May  3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)]
  OS: Windows-10-10.0.19045-SP0
  Lean CLI version: 1.0.178
json_module_handler.get_and_build_module(): non-interactive: essential_properties_value with module Polygon: {}
json_module_handler.get_and_build_module(): non-interactive: required_properties_value with module Polygon: {'polygon-api-key': 'XXX'}
LeanConfigConfigurer.ensure_module_installed(): installing module for module PolygonDataFeed: 306
--> POST https://www.quantconnect.com/api/v2/modules/list with data:
{
    "productId": "XXX",
    "organizationId": "XXXXXXXXX"
}
Request response:
{"files":["QuantConnect.Polygon.2.5.16107.nupkg","QuantConnect.Polygon.2.5.16111.nupkg","QuantConnect.Polygon.2.5.16118.nupkg","QuantConnect.Polygon.2.5.16120.nupkg","QuantConnect.Polygon.2.5.16121.nupkg","Quan
tConnect.Polygon.2.5.16123.nupkg","QuantConnect.Polygon.2.5.16127.nupkg","QuantConnect.Polygon.2.5.16128.nupkg","QuantConnect.Polygon.2.5.16130.nupkg","QuantConnect.Polygon.2.5.16131.nupkg","QuantConnect.Polygo
n.2.5.16133.nupkg","QuantConnect.Polygon.2.5.16135.nupkg","QuantConnect.Polygon.2.5.16138.nupkg","QuantConnect.Polygon.2.5.16140.nupkg","QuantConnect.Polygon.2.5.16142.nupkg","QuantConnect.Polygon.2.5.16144.nup
kg","QuantConnect.Polygon.2.5.16152.nupkg","QuantConnect.Polygon.2.5.16155.nupkg","QuantConnect.Polygon.2.5.16158.nupkg","QuantConnect.Polygon.2.5.16159.nupkg","QuantConnect.Polygon.2.5.16161.nupkg","QuantConne
ct.Polygon.2.5.16163.nupkg","QuantConnect.Polygon.2.5.16164.nupkg","QuantConnect.Polygon.2.5.16165.nupkg","QuantConnect.Polygon.2.5.16167.nupkg","QuantConnect.Polygon.2.5.16168.nupkg","QuantConnect.Polygon.2.5.
16169.nupkg","QuantConnect.Polygon.2.5.16173.nupkg","QuantConnect.Polygon.2.5.16175.nupkg","QuantConnect.Polygon.2.5.16178.nupkg","QuantConnect.Polygon.2.5.16182.nupkg","QuantConnect.Polygon.2.5.16183.nupkg","Q
uantConnect.Polygon.2.5.16187.nupkg"],"success":true}
ModuleManager._download_file(): C:\Users\NickoMR\.lean\modules\QuantConnect.Polygon.2.5.16187.nupkg already exists locally
Traceback (most recent call last):
  File "c:\program files\python38\lib\site-packages\lean\main.py", line 94, in main
    lean.main(standalone_mode=False)
  File "c:\program files\python38\lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
  File "c:\program files\python38\lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\program files\python38\lib\site-packages\lean\click.py", line 159, in invoke
    result = super().invoke(ctx)
  File "c:\program files\python38\lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\program files\python38\lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
  File "c:\program files\python38\lib\site-packages\lean\commands\backtest.py", line 373, in backtest
    data_provider_configurer.configure(lean_config, "backtesting")
  File "c:\program files\python38\lib\site-packages\lean\models\lean_config_configurer.py", line 34, in configure
    self._configure_environment(lean_config, environment_name)
  File "c:\program files\python38\lib\site-packages\lean\models\lean_config_configurer.py", line 55, in _configure_environment
    raise ValueError(f'{self.__class__.__name__} not valid for _configure_environment()')
ValueError: DataProvider not valid for _configure_environment()
Error: DataProvider not valid for _configure_environment()
```

### Lean version
```
D:\source>pip show lean
Name: lean
Version: 1.0.178
Summary: A CLI aimed at making it easier to run QuantConnect's LEAN engine locally and in the cloud
Home-page: https://lean.io/cli
Author: QuantConnect
Author-email: support@quantconnect.com
License: UNKNOWN
Location: c:\program files\python38\lib\site-packages
Requires: requests, maskpass, docker, click, pydantic, python-dateutil, setuptools, quantconnect-stubs, joblib, json5, rich, lxml, wrapt, cryptography
Required-by:
```


See https://github.com/QuantConnect/lean-cli/pull/399

Pull request description:
Fix backtest data provider configuration
An exception was being raised when configuring the data provider from the JSON modules. This was happening when a data provider was specified for the `backtest` command using the `--data-provider` option. 

When the data provider environment is being configured from the JSON modules, the data provider configuration is picked up by `JsonModule.get_configurations_env_values_from_name` but `LeanConfigConfigurer._configure_environment` was not able to handle that type of configuration.

Two things were done:

1. `JsonModule.get_configurations_env_values_from_name` just gets the available environment from the JSON module configurations, which is always one and does not depend on a name.
2. `LeanConfigConfigurer._configure_environment` can just skip data providers instead of raising an exception.

Closes #401

Tested locally with the following:

- Interactive Live Deploy.
- Non Interactive Live Deploy.
- Non Interactive Live Deploy using `--environment` option:
    - Having different environments in `lean.json` to make sure each one is used. For instance having "live-interactive" to deploy to IB and "live-interactive-polygon" (with Polygon as data queue handler and history provider) to deploy to IB using Polygon as data feed. 
- Backtesting and research specifing the data provider to use.

Patch:
diff --git a/lean/commands/live/deploy.py b/lean/commands/live/deploy.py
index baf0d2d..a8f3586 100644
--- a/lean/commands/live/deploy.py
+++ b/lean/commands/live/deploy.py
@@ -54,9 +54,13 @@ def _get_configurable_modules_from_environment(lean_config: Dict[str, Any], envi
 
     brokerage = environment["live-mode-brokerage"]
     data_queue_handlers = environment["data-queue-handler"]
-    [brokerage_configurer] = [local_brokerage for local_brokerage in all_local_brokerages if _get_brokerage_base_name(local_brokerage.get_live_name(environment_name)) == _get_brokerage_base_name(brokerage)]
+    [brokerage_configurer] = [local_brokerage
+                              for local_brokerage in all_local_brokerages
+                              if _get_brokerage_base_name(local_brokerage.get_live_name()) == _get_brokerage_base_name(brokerage)]
     data_queue_handlers_base_names = [_get_brokerage_base_name(data_queue_handler) for data_queue_handler in data_queue_handlers]
-    data_feed_configurers = [local_data_feed for local_data_feed in all_local_data_feeds if _get_brokerage_base_name(local_data_feed.get_live_name(environment_name)) in data_queue_handlers_base_names]
+    data_feed_configurers = [local_data_feed
+                             for local_data_feed in all_local_data_feeds
+                             if _get_brokerage_base_name(local_data_feed.get_live_name()) in data_queue_handlers_base_names]
     return brokerage_configurer, data_feed_configurers
 
 
@@ -335,7 +339,7 @@ def deploy(project: Path,
     if environment is not None:
         environment_name = environment
         lean_config = lean_config_manager.get_complete_lean_config(environment_name, algorithm_file, None)
-        
+
         lean_environment = lean_config["environments"][environment_name]
         for key in ["live-mode-brokerage", "data-queue-handler"]:
             if key not in lean_environment:
@@ -362,7 +366,7 @@ def deploy(project: Path,
                         kwargs[property_name_to_fill] = property_value_to_fill
                         lean_config[condition._dependent_config_id] = property_value_to_fill
                     break
-        
+
         for local_data_feed in all_local_data_feeds:
             configuration_environments: List[ConfigurationsEnvConfiguration] = [config for config in local_data_feed._lean_configs if config._is_type_configurations_env]
             for configuration_environment in configuration_environments:
diff --git a/lean/models/brokerages/local/data_feed.py b/lean/models/brokerages/local/data_feed.py
index ec5feac..34be975 100644
--- a/lean/models/brokerages/local/data_feed.py
+++ b/lean/models/brokerages/local/data_feed.py
@@ -21,11 +21,9 @@ class DataFeed(LeanConfigConfigurer):
     def __init__(self, json_datafeed_data: Dict[str, Any]) -> None:
         super().__init__(json_datafeed_data)
 
-    def get_live_name(self, environment_name: str) -> str:
+    def get_live_name(self) -> str:
         live_name = self._id
-        environment_obj = self.get_configurations_env_values_from_name(
-            environment_name)
+        environment_obj = self.get_configurations_env_values()
         if environment_obj:
-            [live_name] = [x["value"]
-                           for x in environment_obj if x["name"] == "data-queue-handler"]
+            [live_name] = [x["value"] for x in environment_obj if x["name"] == "data-queue-handler"]
         return live_name
diff --git a/lean/models/brokerages/local/local_brokerage.py b/lean/models/brokerages/local/local_brokerage.py
index d1743ba..c22f7fe 100644
--- a/lean/models/brokerages/local/local_brokerage.py
+++ b/lean/models/brokerages/local/local_brokerage.py
@@ -21,11 +21,9 @@ class LocalBrokerage(LeanConfigConfigurer):
     def __init__(self, json_brokerage_data: Dict[str, Any]) -> None:
         super().__init__(json_brokerage_data)
 
-    def get_live_name(self, environment_name: str) -> str:
+    def get_live_name(self) -> str:
         live_name = self._id
-        environment_obj = self.get_configurations_env_values_from_name(
-            environment_name)
+        environment_obj = self.get_configurations_env_values()
         if environment_obj:
-            [live_name] = [x["value"]
-                           for x in environment_obj if x["name"] == "live-mode-brokerage"]
+            [live_name] = [x["value"] for x in environment_obj if x["name"] == "live-mode-brokerage"]
         return live_name
diff --git a/lean/models/json_module.py b/lean/models/json_module.py
index c27ce36..41ee996 100644
--- a/lean/models/json_module.py
+++ b/lean/models/json_module.py
@@ -85,13 +85,14 @@ class JsonModule(ABC):
         for key, value in key_and_values.items():
             self.update_value_for_given_config(key, value)
 
-    def get_configurations_env_values_from_name(self, target_env: str) -> List[Dict[str, str]]:
+    def get_configurations_env_values(self) -> List[Dict[str, str]]:
         env_config_values = []
         [env_config] = [config for config in self._lean_configs if
                         config._is_type_configurations_env and self.check_if_config_passes_filters(
                             config)
                         ] or [None]
         if env_config is not None:
+            # Always get the first one, since we only expect one env config in the json modules file
             env_config_values = list(env_config._env_and_values.values())[0]
         return env_config_values
 
diff --git a/lean/models/lean_config_configurer.py b/lean/models/lean_config_configurer.py
index c8a560e..7ff8f8f 100644
--- a/lean/models/lean_config_configurer.py
+++ b/lean/models/lean_config_configurer.py
@@ -39,7 +39,7 @@ class LeanConfigConfigurer(JsonModule, ABC):
         :param lean_config: the Lean configuration dict to write to
         :param environment_name: the name of the environment to update
         """
-        for environment_config in self.get_configurations_env_values_from_name(environment_name):
+        for environment_config in self.get_configurations_env_values():
             environment_config_name = environment_config["name"]
             if self.__class__.__name__ == 'DataFeed':
                 if environment_config_name == "data-queue-handler":
@@ -51,8 +51,6 @@ class LeanConfigConfigurer(JsonModule, ABC):
             elif self.__class__.__name__ == 'LocalBrokerage':
                 if environment_config_name != "data-queue-handler":
                     lean_config["environments"][environment_name][environment_config_name] = environment_config["value"]
-            else:
-                raise ValueError(f'{self.__class__.__name__} not valid for _configure_environment()')
 
     def configure_credentials(self, lean_config: Dict[str, Any]) -> None:
         """Configures the credentials in the Lean config for this brokerage and saves them persistently to disk.


Test patch:
diff --git a/tests/models/test_lean_config_configurer.py b/tests/models/test_lean_config_configurer.py
new file mode 100644
index 0000000..dccfd93
--- /dev/null
+++ b/tests/models/test_lean_config_configurer.py
@@ -0,0 +1,123 @@
+# QUANTCONNECT.COM - Democratizing Finance, Empowering Individuals.
+# Lean CLI v1.0. Copyright 2021 QuantConnect Corporation.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import json
+from typing import Dict, Any
+from unittest import mock
+
+from lean.models.brokerages.local import DataFeed
+from lean.models.data_providers import DataProvider
+from lean.models.lean_config_configurer import LeanConfigConfigurer
+
+JSON_MODULE = json.loads("""
+{
+    "type": [
+        "data-queue-handler",
+        "data-provider"
+    ],
+    "product-id": "305",
+    "id": "PolygonDataFeed",
+    "display-id": "Polygon",
+    "installs": true,
+    "configurations": [
+        {
+            "id": "polygon-api-key",
+            "cloud-id": "apiKey",
+            "type": "input",
+            "value": "",
+            "input-method": "prompt",
+            "prompt-info": "Your Polygon.io API Key",
+            "help": "Your Polygon.io API Key"
+        },
+        {
+            "id": "environments",
+            "type": "configurations-env",
+            "value": [
+                {
+                    "name": "lean-cli",
+                    "value": [
+                        {
+                            "name": "data-queue-handler",
+                            "value": "QuantConnect.Polygon.PolygonDataQueueHandler"
+                        },
+                        {
+                            "name": "history-provider",
+                            "value": [
+                                "QuantConnect.Polygon.PolygonDataQueueHandler",
+                                "SubscriptionDataReaderHistoryProvider"
+                            ]
+                        }
+                    ]
+                }
+            ]
+        },
+        {
+            "id": "data-provider",
+            "type": "info",
+            "value": "QuantConnect.Lean.Engine.DataFeeds.DownloaderDataProvider"
+        },
+        {
+            "id": "data-downloader",
+            "type": "info",
+            "value": "QuantConnect.Polygon.PolygonDataDownloader"
+        }
+    ]
+}
+""")
+
+
+def test_gets_environment_from_configuration() -> None:
+    module = LeanConfigConfigurer(JSON_MODULE)
+    environment_values = module.get_configurations_env_values()
+
+    assert environment_values == JSON_MODULE["configurations"][1]["value"][0]["value"]
+
+
+def get_lean_config() -> Dict[str, Any]:
+    return {
+        "environments": {
+            "live-ib-polygon": {
+                "live-mode": True,
+                "live-mode-brokerage": "InteractiveBrokersBrokerage",
+                "setup-handler": "QuantConnect.Lean.Engine.Setup.BrokerageSetupHandler",
+                "result-handler": "QuantConnect.Lean.Engine.Results.LiveTradingResultHandler",
+                "data-feed-handler": "QuantConnect.Lean.Engine.DataFeeds.LiveTradingDataFeed",
+                "data-queue-handler": ["QuantConnect.Brokerages.InteractiveBrokers.InteractiveBrokersBrokerage"],
+                "real-time-handler": "QuantConnect.Lean.Engine.RealTime.LiveTradingRealTimeHandler",
+                "transaction-handler": "QuantConnect.Lean.Engine.TransactionHandlers.BrokerageTransactionHandler",
+                "history-provider": [
+                    "BrokerageHistoryProvider",
+                    "SubscriptionDataReaderHistoryProvider"
+                ]
+            }
+        }
+    }
+
+
+def test_configures_environment_with_module() -> None:
+    with mock.patch.object(DataFeed, "configure_credentials"):
+        lean_config = get_lean_config()
+        module = DataFeed(JSON_MODULE)
+        module.configure(lean_config, "live-ib-polygon")
+
+        assert lean_config != get_lean_config()
+        assert "QuantConnect.Polygon.PolygonDataQueueHandler" in lean_config["environments"]["live-ib-polygon"]["data-queue-handler"]
+
+
+def test_invalid_environment_configuration_is_ignored() -> None:
+    with mock.patch.object(DataProvider, "configure_credentials"):
+        lean_config = get_lean_config()
+        module = DataProvider(JSON_MODULE)
+        module.configure(lean_config, "live-ib-polygon")
+
+        assert lean_config == get_lean_config()