Repository: freqtrade/freqtrade

Issue:
Test Pairlist / precision filter problem
<!-- 
Have you searched for similar issues before posting it?

similar to #3363
-->

## Describe your environment

  * Operating system: Ubuntu 18.04
  * Python Version: 3.7.4
  * CCXT version: 1.29.5
  * Freqtrade Version: freqtrade develop-aff80d73

  
Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.

## Describe the problem:

precision filter still makes trouble ...

### Steps to reproduce:

relevant part of the config:

```
    "pairlists": [
        
        {
            "method": "VolumePairList",
            "number_assets": 30,
            "sort_key": "quoteVolume",
            "refresh_period": 1800
        },
        {"method": "PrecisionFilter"},
        {"method": "PriceFilter", "low_price_ratio": 0.01},
        {"method": "SpreadFilter", "max_spread_ratio": 0.0075}


```


### Observed Results:

see log below, if precision filter is removed - test pairlist works fine

### Relevant code exceptions or logs

Note: Please copy/paste text of the messages, no screenshots of logs please.

  ```
(.env) (base) andreas@andreas-G5-5590:~/freqtrade$ freqtrade test-pairlist --config config_ETHpairs2.json --quote  ETH
2020-06-09 12:26:48,684 - freqtrade.loggers - INFO - Verbosity set to 0
2020-06-09 12:26:48,684 - freqtrade.configuration.configuration - INFO - Using user-data directory: /home/andreas/freqtrade/user_data ...
2020-06-09 12:26:48,684 - freqtrade.configuration.configuration - INFO - Using data directory: /home/andreas/freqtrade/user_data/data/binance ...
2020-06-09 12:26:48,684 - freqtrade.configuration.check_exchange - INFO - Checking exchange...
2020-06-09 12:26:48,684 - freqtrade.configuration.check_exchange - INFO - Exchange "binance" is officially supported by the Freqtrade development team.
2020-06-09 12:26:48,684 - freqtrade.configuration.configuration - INFO - Using pairlist from configuration.
2020-06-09 12:26:48,684 - freqtrade.configuration.config_validation - INFO - Validating configuration ...
2020-06-09 12:26:48,685 - freqtrade.exchange.exchange - INFO - Instance is running with dry_run enabled
2020-06-09 12:26:48,685 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True}
2020-06-09 12:26:48,688 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True, 'rateLimit': 5000}
2020-06-09 12:26:48,691 - freqtrade.exchange.exchange - INFO - Using Exchange "Binance"
2020-06-09 12:26:48,691 - freqtrade.resolvers.exchange_resolver - INFO - Using resolved exchange 'Binance'...
2020-06-09 12:26:48,697 - freqtrade.resolvers.iresolver - INFO - Using resolved pairlist VolumePairList from '/home/andreas/freqtrade/freqtrade/pairlist/VolumePairList.py'...
2020-06-09 12:26:48,698 - freqtrade.resolvers.iresolver - INFO - Using resolved pairlist PrecisionFilter from '/home/andreas/freqtrade/freqtrade/pairlist/PrecisionFilter.py'...
2020-06-09 12:26:48,698 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/andreas/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/home/andreas/freqtrade/freqtrade/commands/pairlist_commands.py", line 29, in start_test_pairlist
    pairlists = PairListManager(exchange, config)
  File "/home/andreas/freqtrade/freqtrade/pairlist/pairlistmanager.py", line 38, in __init__
    pairlist_pos=len(self._pairlist_handlers)
  File "/home/andreas/freqtrade/freqtrade/resolvers/pairlist_resolver.py", line 42, in load_pairlist
    'pairlist_pos': pairlist_pos},
  File "/home/andreas/freqtrade/freqtrade/resolvers/iresolver.py", line 136, in load_object
    kwargs=kwargs)
  File "/home/andreas/freqtrade/freqtrade/resolvers/iresolver.py", line 113, in _load_object
    return module(**kwargs)
  File "/home/andreas/freqtrade/freqtrade/pairlist/PrecisionFilter.py", line 20, in __init__
    self._stoploss = self._config['stoploss']
KeyError: 'stoploss'

  ```

Backtesting Error (after update)
## Step 1: Have you search for this issue before posting it?



## Step 2: Describe your environment

  * Operating system: Ubuntu 18.04
  * Python Version: 3.7.4
  * CCXT version: ccxt==1.27.91
  * Branch: Develop
  * Last Commit ID: b50d072d824339110e1957c804d3f47766ac9a2a

  
## Step 3: Describe the problem:

backtesting not working after update in Dev branch

### Steps to reproduce:

1. backtest done, okay
2. update on latest dev version
3. same CLI command as before, backtesting not working anymore

  
### Observed Results:

 see log below

### Relevant code exceptions or logs:

  ```
  freqtrade backtesting  --strategy-list ADXMomentum AdxSmas ASDTSRockwellTrading AverageStrategy AwesomeMacd BbandRsi BinHV27 BinHV45 CCIStrategy ClucMay72018 CMCWinner CofiBitStrategy CombinedBinHAndCluc EMASkipPump Low_BB MACDStrategy MACDStrategy_crossed MultiRSI Quickie ReinforcedQuickie ReinforcedSmoothScalp Scalp Simple SmoothScalp Ichimoku BbandRsi_AP4 Bblow_AP Bblow_AP1 --config config_pairs1.json  --timerange=20180801-20200515 --export trades --ticker-interval 1h
2020-05-23 13:10:49,365 - freqtrade.loggers - INFO - Verbosity set to 0
2020-05-23 13:10:49,365 - freqtrade.configuration.configuration - INFO - Parameter -i/--ticker-interval detected ... Using ticker_interval: 1h ...
2020-05-23 13:10:49,365 - freqtrade.configuration.configuration - INFO - Using max_open_trades: 10 ...
2020-05-23 13:10:49,365 - freqtrade.configuration.configuration - INFO - Parameter --timerange detected: 20180801-20200515 ...
2020-05-23 13:10:49,365 - freqtrade.configuration.configuration - INFO - Using user-data directory: /home/andreas/freqtrade/user_data ...
2020-05-23 13:10:49,365 - freqtrade.configuration.configuration - INFO - Using data directory: /home/andreas/freqtrade/user_data/data/binance ...
2020-05-23 13:10:49,365 - freqtrade.configuration.configuration - INFO - Using strategy list of 28 strategies
2020-05-23 13:10:49,365 - freqtrade.configuration.configuration - INFO - Overriding ticker interval with Command line argument
2020-05-23 13:10:49,365 - freqtrade.configuration.configuration - INFO - Parameter --export detected: trades ...
2020-05-23 13:10:49,366 - freqtrade.configuration.check_exchange - INFO - Checking exchange...
2020-05-23 13:10:49,366 - freqtrade.configuration.check_exchange - INFO - Exchange "binance" is officially supported by the Freqtrade development team.
2020-05-23 13:10:49,366 - freqtrade.configuration.configuration - INFO - Using pairlist from configuration.
2020-05-23 13:10:49,366 - freqtrade.configuration.config_validation - INFO - Validating configuration ...
2020-05-23 13:10:49,367 - freqtrade.commands.optimize_commands - INFO - Starting freqtrade in Backtesting mode
2020-05-23 13:10:49,367 - freqtrade.exchange.exchange - INFO - Instance is running with dry_run enabled
2020-05-23 13:10:49,367 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True}
2020-05-23 13:10:49,369 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True, 'rateLimit': 5000}
2020-05-23 13:10:49,371 - freqtrade.exchange.exchange - INFO - Using Exchange "Binance"
2020-05-23 13:10:50,218 - freqtrade.resolvers.exchange_resolver - INFO - Using resolved exchange 'Binance'...
2020-05-23 13:10:50,219 - freqtrade.resolvers.iresolver - INFO - Using resolved pairlist StaticPairList from '/home/andreas/freqtrade/freqtrade/pairlist/StaticPairList.py'...
2020-05-23 13:10:50,219 - freqtrade.resolvers.iresolver - INFO - Using resolved pairlist PrecisionFilter from '/home/andreas/freqtrade/freqtrade/pairlist/PrecisionFilter.py'...
2020-05-23 13:10:50,220 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/andreas/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/home/andreas/freqtrade/freqtrade/commands/optimize_commands.py", line 48, in start_backtesting
    backtesting = Backtesting(config)
  File "/home/andreas/freqtrade/freqtrade/optimize/backtesting.py", line 67, in __init__
    self.pairlists = PairListManager(self.exchange, self.config)
  File "/home/andreas/freqtrade/freqtrade/pairlist/pairlistmanager.py", line 38, in __init__
    pairlist_pos=len(self._pairlist_handlers)
  File "/home/andreas/freqtrade/freqtrade/resolvers/pairlist_resolver.py", line 42, in load_pairlist
    'pairlist_pos': pairlist_pos},
  File "/home/andreas/freqtrade/freqtrade/resolvers/iresolver.py", line 136, in load_object
    kwargs=kwargs)
  File "/home/andreas/freqtrade/freqtrade/resolvers/iresolver.py", line 113, in _load_object
    return module(**kwargs)
  File "/home/andreas/freqtrade/freqtrade/pairlist/PrecisionFilter.py", line 20, in __init__
    self._stoploss = self._config['stoploss']
KeyError: 'stoploss'

  ```

Test Pairlist / precision filter problem
<!-- 
Have you searched for similar issues before posting it?

similar to #3363
-->

## Describe your environment

  * Operating system: Ubuntu 18.04
  * Python Version: 3.7.4
  * CCXT version: 1.29.5
  * Freqtrade Version: freqtrade develop-aff80d73

  
Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.

## Describe the problem:

precision filter still makes trouble ...

### Steps to reproduce:

relevant part of the config:

```
    "pairlists": [
        
        {
            "method": "VolumePairList",
            "number_assets": 30,
            "sort_key": "quoteVolume",
            "refresh_period": 1800
        },
        {"method": "PrecisionFilter"},
        {"method": "PriceFilter", "low_price_ratio": 0.01},
        {"method": "SpreadFilter", "max_spread_ratio": 0.0075}


```


### Observed Results:

see log below, if precision filter is removed - test pairlist works fine

### Relevant code exceptions or logs

Note: Please copy/paste text of the messages, no screenshots of logs please.

  ```
(.env) (base) andreas@andreas-G5-5590:~/freqtrade$ freqtrade test-pairlist --config config_ETHpairs2.json --quote  ETH
2020-06-09 12:26:48,684 - freqtrade.loggers - INFO - Verbosity set to 0
2020-06-09 12:26:48,684 - freqtrade.configuration.configuration - INFO - Using user-data directory: /home/andreas/freqtrade/user_data ...
2020-06-09 12:26:48,684 - freqtrade.configuration.configuration - INFO - Using data directory: /home/andreas/freqtrade/user_data/data/binance ...
2020-06-09 12:26:48,684 - freqtrade.configuration.check_exchange - INFO - Checking exchange...
2020-06-09 12:26:48,684 - freqtrade.configuration.check_exchange - INFO - Exchange "binance" is officially supported by the Freqtrade development team.
2020-06-09 12:26:48,684 - freqtrade.configuration.configuration - INFO - Using pairlist from configuration.
2020-06-09 12:26:48,684 - freqtrade.configuration.config_validation - INFO - Validating configuration ...
2020-06-09 12:26:48,685 - freqtrade.exchange.exchange - INFO - Instance is running with dry_run enabled
2020-06-09 12:26:48,685 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True}
2020-06-09 12:26:48,688 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True, 'rateLimit': 5000}
2020-06-09 12:26:48,691 - freqtrade.exchange.exchange - INFO - Using Exchange "Binance"
2020-06-09 12:26:48,691 - freqtrade.resolvers.exchange_resolver - INFO - Using resolved exchange 'Binance'...
2020-06-09 12:26:48,697 - freqtrade.resolvers.iresolver - INFO - Using resolved pairlist VolumePairList from '/home/andreas/freqtrade/freqtrade/pairlist/VolumePairList.py'...
2020-06-09 12:26:48,698 - freqtrade.resolvers.iresolver - INFO - Using resolved pairlist PrecisionFilter from '/home/andreas/freqtrade/freqtrade/pairlist/PrecisionFilter.py'...
2020-06-09 12:26:48,698 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/andreas/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/home/andreas/freqtrade/freqtrade/commands/pairlist_commands.py", line 29, in start_test_pairlist
    pairlists = PairListManager(exchange, config)
  File "/home/andreas/freqtrade/freqtrade/pairlist/pairlistmanager.py", line 38, in __init__
    pairlist_pos=len(self._pairlist_handlers)
  File "/home/andreas/freqtrade/freqtrade/resolvers/pairlist_resolver.py", line 42, in load_pairlist
    'pairlist_pos': pairlist_pos},
  File "/home/andreas/freqtrade/freqtrade/resolvers/iresolver.py", line 136, in load_object
    kwargs=kwargs)
  File "/home/andreas/freqtrade/freqtrade/resolvers/iresolver.py", line 113, in _load_object
    return module(**kwargs)
  File "/home/andreas/freqtrade/freqtrade/pairlist/PrecisionFilter.py", line 20, in __init__
    self._stoploss = self._config['stoploss']
KeyError: 'stoploss'

  ```



Discussion:

after looking at this a bit, i think we must restrict backtesting to StaticPairlist - at least if `--strategy-list` is used.

Otherwise, results will not be comparable / reproducible, as each strategy may result in a different pairlist, so you'd be comparing one strategy running on 20 pairs with another strategy running on 5 pairs.

Stoploss is just a symptom here ... 

I can solve it for single strategies easily - but i don't think there should be a solution for multiple strategies as to handle that correctly (identical behaviour to running backtesting with `--strategy` multiple times) we'd have to recreate the pairlist for every strategy - but that has the side-effect of having non-comparable results (different pairs used ...).

I think for that case it may be best to use `freqtrade test-pairlist` - and then use StaticPairlist instead.
okay, I see the point you are making - but I dont understand how this happens. My understanding is the following:

1.) I got one config in which I use all filters (except volume) on top of my static pair-list
2.) result will be a dynamic pair-list on which all strategies in my list will be backtested

of course this is a snapshot, but all strategies in the list are backtested based on the same (dynamic pairlist), hence they are comparable, right?

or is FT creating a new pair list between each backtest? in this case results would of course be not comparable...
The problem is that your configuration does not contain a stoploss (which is valid) - but is using the stoploss from the strategy - which may or may not be different between the strategies.

Now PrecisionFilter uses this stoploss value - as it's checking if you're allowed to create a stoploss below the entry price (so it needs to know this stoploss value).

The strategy-list logic does not write this stoploss value back to the global config (otherwise it would be set identical for all strategies), but keeps it in it's local config.

A quick fix is to disable PrecisionFilter automatically when it's not used (that's how it worked before) - but it'll only hide the problem, and it'll not be obvious that it was disabled, nor why.
okay, got you - clear now ... means precision filter needs to be removed from the global config if working with the strategy list CLI command

merci!
well that'll fix it short term for you - yes - but it won't fix the real problem 😆 
I know this problem (stoploss is saved in the pairlist handler attribute and is not refreshed when a new strategy from the list is loaded) and I was looking at it while cleaning up/refactoring the pairlist handlers, but decided to leave it as it is to keep the current behavior... (but also maybe it's me who introduced it in one of the commits, need to compare with the previous master closer)

yes you did - but because i asked you to ... i remember that line and almost exactly my comment ... 
but it's not relevant - the previous behaviour (disabling PrecisionFilter) was not better - this at least exposed this problematic behaviour.
It can be fixed easily -- simply do not fetch and save into the pairlist class attributes anything from configuration in the `__init__` method, instead, fetch values from configuration (stoploss, in this case) later in `gen_pairlist()` and in `filter_pairlist()`...
not true - backtesting updates the pairlist before loading any strategy.

As mentioned above - to handle this the same way than when backtesting one strategy, we'd need to refresh the pairlist between every strategy - however that would defeat the purpose of `--strategy-list` - as it'll make the results non-comparable.

> The problem is that your configuration does not contain a stoploss (which is valid) - but is using the stoploss from the strategy - which may or may not be different between the strategies.
> 
> Now PrecisionFilter uses this stoploss value - as it's checking if you're allowed to create a stoploss below the entry price (so it needs to know this stoploss value).
> 
> The strategy-list logic does not write this stoploss value back to the global config (otherwise it would be set identical for all strategies), but keeps it in it's local config.
> 
> A quick fix is to disable PrecisionFilter automatically when it's not used (that's how it worked before) - but it'll only hide the problem, and it'll not be obvious that it was disabled, nor why.

I did not fully understand. Is it a problem caused by the person using this bot?
it's a configuration problem - if you use StaticPairList for backtesting, then it'll work fine. 



Resolving patch:
diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -65,20 +65,6 @@ def __init__(self, config: Dict[str, Any]) -> None:
         self.strategylist: List[IStrategy] = []
         self.exchange = ExchangeResolver.load_exchange(self.config['exchange']['name'], self.config)
 
-        self.pairlists = PairListManager(self.exchange, self.config)
-        if 'VolumePairList' in self.pairlists.name_list:
-            raise OperationalException("VolumePairList not allowed for backtesting.")
-
-        self.pairlists.refresh_pairlist()
-
-        if len(self.pairlists.whitelist) == 0:
-            raise OperationalException("No pair in whitelist.")
-
-        if config.get('fee'):
-            self.fee = config['fee']
-        else:
-            self.fee = self.exchange.get_fee(symbol=self.pairlists.whitelist[0])
-
         if self.config.get('runmode') != RunMode.HYPEROPT:
             self.dataprovider = DataProvider(self.config, self.exchange)
             IStrategy.dp = self.dataprovider
@@ -101,6 +87,25 @@ def __init__(self, config: Dict[str, Any]) -> None:
         self.timeframe = str(self.config.get('ticker_interval'))
         self.timeframe_min = timeframe_to_minutes(self.timeframe)
 
+        self.pairlists = PairListManager(self.exchange, self.config)
+        if 'VolumePairList' in self.pairlists.name_list:
+            raise OperationalException("VolumePairList not allowed for backtesting.")
+
+        if len(self.strategylist) > 1 and 'PrecisionFilter' in self.pairlists.name_list:
+            raise OperationalException(
+                "PrecisionFilter not allowed for backtesting multiple strategies."
+            )
+
+        self.pairlists.refresh_pairlist()
+
+        if len(self.pairlists.whitelist) == 0:
+            raise OperationalException("No pair in whitelist.")
+
+        if config.get('fee'):
+            self.fee = config['fee']
+        else:
+            self.fee = self.exchange.get_fee(symbol=self.pairlists.whitelist[0])
+
         # Get maximum required startup period
         self.required_startup = max([strat.startup_candle_count for strat in self.strategylist])
         # Load one (first) strategy
diff --git a/freqtrade/pairlist/PrecisionFilter.py b/freqtrade/pairlist/PrecisionFilter.py
--- a/freqtrade/pairlist/PrecisionFilter.py
+++ b/freqtrade/pairlist/PrecisionFilter.py
@@ -5,7 +5,7 @@
 from typing import Any, Dict
 
 from freqtrade.pairlist.IPairList import IPairList
-
+from freqtrade.exceptions import OperationalException
 
 logger = logging.getLogger(__name__)
 
@@ -17,6 +17,10 @@ def __init__(self, exchange, pairlistmanager,
                  pairlist_pos: int) -> None:
         super().__init__(exchange, pairlistmanager, config, pairlistconfig, pairlist_pos)
 
+        if 'stoploss' not in self._config:
+            raise OperationalException(
+                'PrecisionFilter can only work with stoploss defined. Please add the '
+                'stoploss key to your configuration (overwrites eventual strategy settings).')
         self._stoploss = self._config['stoploss']
         self._enabled = self._stoploss != 0
 


Test patch:
diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py
--- a/tests/optimize/test_backtesting.py
+++ b/tests/optimize/test_backtesting.py
@@ -401,6 +401,38 @@ def test_backtesting_no_pair_left(default_conf, mocker, caplog, testdatadir) ->
         Backtesting(default_conf)
 
 
+def test_backtesting_pairlist_list(default_conf, mocker, caplog, testdatadir, tickers) -> None:
+    mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True))
+    mocker.patch('freqtrade.exchange.Exchange.get_tickers', tickers)
+    mocker.patch('freqtrade.exchange.Exchange.price_to_precision', lambda s, x, y: y)
+    mocker.patch('freqtrade.data.history.get_timerange', get_timerange)
+    patch_exchange(mocker)
+    mocker.patch('freqtrade.optimize.backtesting.Backtesting.backtest')
+    mocker.patch('freqtrade.pairlist.pairlistmanager.PairListManager.whitelist',
+                 PropertyMock(return_value=['XRP/BTC']))
+    mocker.patch('freqtrade.pairlist.pairlistmanager.PairListManager.refresh_pairlist')
+
+    default_conf['ticker_interval'] = "1m"
+    default_conf['datadir'] = testdatadir
+    default_conf['export'] = None
+    # Use stoploss from strategy
+    del default_conf['stoploss']
+    default_conf['timerange'] = '20180101-20180102'
+
+    default_conf['pairlists'] = [{"method": "VolumePairList", "number_assets": 5}]
+    with pytest.raises(OperationalException, match='VolumePairList not allowed for backtesting.'):
+        Backtesting(default_conf)
+
+    default_conf['pairlists'] = [{"method": "StaticPairList"}, {"method": "PrecisionFilter"}, ]
+    Backtesting(default_conf)
+
+    # Multiple strategies
+    default_conf['strategy_list'] = ['DefaultStrategy', 'TestStrategyLegacy']
+    with pytest.raises(OperationalException,
+                       match='PrecisionFilter not allowed for backtesting multiple strategies.'):
+        Backtesting(default_conf)
+
+
 def test_backtest(default_conf, fee, mocker, testdatadir) -> None:
     default_conf['ask_strategy']['use_sell_signal'] = False
     mocker.patch('freqtrade.exchange.Exchange.get_fee', fee)
diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py
--- a/tests/pairlist/test_pairlist.py
+++ b/tests/pairlist/test_pairlist.py
@@ -362,6 +362,17 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t
                     assert not log_has(logmsg, caplog)
 
 
+def test_PrecisionFilter_error(mocker, whitelist_conf, tickers) -> None:
+    whitelist_conf['pairlists'] = [{"method": "StaticPairList"}, {"method": "PrecisionFilter"}]
+    del whitelist_conf['stoploss']
+
+    mocker.patch('freqtrade.exchange.Exchange.exchange_has', MagicMock(return_value=True))
+
+    with pytest.raises(OperationalException,
+                       match=r"PrecisionFilter can only work with stoploss defined\..*"):
+        PairListManager(MagicMock, whitelist_conf)
+
+
 def test_gen_pair_whitelist_not_supported(mocker, default_conf, tickers) -> None:
     default_conf['pairlists'] = [{'method': 'VolumePairList', 'number_assets': 10}]