Repository: freqtrade/freqtrade

Issue:
Problems with Binance UP/DOWN tokens
<!-- 
Have you searched for similar issues before posting it?

If you have discovered a bug in the bot, please [search our issue tracker](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue). 
If it hasn't been reported, please create a new issue.

Please do not use bug reports to request new features.
-->

## Describe your environment

  * Operating system: Ubuntu 20.04
  * Python Version:  3.8.5
  * CCXT version:  1.40.99
  * Freqtrade Version: develop-3a834929

  
Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.

## Describe the problem:

pair list is "loosing" UP / DOWN token

### Steps to reproduce:

following pairlist setup in the config:

1) all UP / DOWN token being white-listed

2) following filter

    "pairlists": [

        {"method": "StaticPairList"},        
        {"method": "AgeFilter", "min_days_listed": 10},        
        {"method": "PrecisionFilter"},
        {"method": "PriceFilter", "low_price_ratio": 0.01},
        {"method": "SpreadFilter", "max_spread_ratio": 0.0075},
        {"method": "ShuffleFilter", "seed": 42}
     ],

  
### Observed Results:

  * if the Bot is being started almost all pairs out if the white list are available:
```
Using whitelist ['StaticPairList', 'AgeFilter', 'PrecisionFilter', 'PriceFilter', 'SpreadFilter', 'ShuffleFilter'] with 28 pairs
XTZDOWN/USDT, SXPUP/USDT, TRXDOWN/USDT, DOTUP/USDT, AAVEUP/USDT, LINKDOWN/USDT, EOSUP/USDT, XRPUP/USDT, FILUP/USDT, XTZUP/USDT, DOTDOWN/USDT, BNBDOWN/USDT, YFIUP/USDT, ETHUP/USDT, LTCUP/USDT, FILDOWN/USDT, BTCUP/USDT, BTCDOWN/USDT, SUSHIUP/USDT, ETHDOWN/USDT, XRPDOWN/USDT, UNIUP/USDT, BNBUP/USDT, ADAUP/USDT, TRXUP/USDT, LINKUP/USDT, EOSDOWN/USDT, LTCDOWN/USDT
```

  * after a while the Bot is "loosing" almost all pairs until he has only a few left, this behaviour I have not seen so far. I am running the same bot with "standard" USDT pairs, no issues here

```
Using whitelist ['StaticPairList', 'AgeFilter', 'PrecisionFilter', 'PriceFilter', 'SpreadFilter', 'ShuffleFilter'] with 6 pairs
LINKUP/USDT, SXPUP/USDT, BTCDOWN/USDT, ETHUP/USDT, XRPUP/USDT, YFIUP/USDT

```

### Relevant code exceptions or logs

the logs are looking good, nothing suspicious

  ```
2021-01-29 14:00:00,791 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:00:16,644 - freqtrade.data.converter - INFO - Missing data fillup for XRPUP/USDT: before: 999 - after: 1003
2021-01-29 14:00:16,665 - freqtrade.data.converter - INFO - Missing data fillup for BTCDOWN/USDT: before: 999 - after: 1003
2021-01-29 14:00:16,685 - freqtrade.data.converter - INFO - Missing data fillup for LTCUP/USDT: before: 999 - after: 1003
2021-01-29 14:00:16,705 - freqtrade.data.converter - INFO - Missing data fillup for YFIUP/USDT: before: 999 - after: 1003
2021-01-29 14:00:16,766 - freqtrade.data.converter - INFO - Missing data fillup for LINKUP/USDT: before: 999 - after: 1003
2021-01-29 14:00:16,809 - freqtrade.data.converter - INFO - Missing data fillup for ETHUP/USDT: before: 999 - after: 1003
2021-01-29 14:00:16,935 - freqtrade.data.converter - INFO - Missing data fillup for EOSUP/USDT: before: 999 - after: 1003
2021-01-29 14:00:16,976 - freqtrade.data.converter - INFO - Missing data fillup for SXPUP/USDT: before: 999 - after: 1003
2021-01-29 14:00:16,998 - freqtrade.data.converter - INFO - Missing data fillup for ETHDOWN/USDT: before: 999 - after: 1003
2021-01-29 14:00:17,019 - freqtrade.data.converter - INFO - Missing data fillup for DOTUP/USDT: before: 999 - after: 1003
2021-01-29 14:00:17,039 - freqtrade.data.converter - INFO - Missing data fillup for EOSDOWN/USDT: before: 999 - after: 1003
2021-01-29 14:00:21,074 - freqtrade.data.converter - INFO - Missing data fillup for XRPUP/USDT: before: 999 - after: 1003
2021-01-29 14:01:04,752 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:02:04,779 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:03:04,806 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:04:04,836 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:05:04,864 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:06:04,890 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:07:04,918 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:08:04,945 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:09:04,972 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:10:04,998 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:11:05,026 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:12:05,055 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:13:05,085 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:14:05,115 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'
2021-01-29 14:15:05,139 - freqtrade.worker - INFO - Bot heartbeat. PID=792, version='develop-3a834929', state='RUNNING'

  ```



Discussion:
That's what `filter` is used for, I don't knew what do you expect. But when a pair is filtered, it is removed from the whitelist.
I think you need debug logging level if you want to see log of when pairs are removed
i don't think with this section of the log we'll find anything.

Debug-log should not be required - however, whenever a filter removes a pair, this is logged (so it's clear which filter removes the pair). That's not included, so further investigation will not be possible.
@mobrine1  ... yes and no - yes, it should filter but it should not remove all pairs step by step ... normally this kind of bot and pairlist config removes in average 10-20% of the whitelisted pairs (mainly due to spread filter), but usually spread goes back below 0,75% and it would put the pair back to the white list, thus over time the whitelisted pairs are constantly fluctuating around one average value and not going down to zero.

@xmatthias ... pls find enclosed the whole log file for further investigations. btw after re-starting the bot he got a full pairlist (apart from the ones being filtered out immediately after start-up)
[freqtrade-service-5.zip](https://github.com/freqtrade/freqtrade/files/5897664/freqtrade-service-5.zip)


Resolving patch:
diff --git a/freqtrade/plugins/pairlist/IPairList.py b/freqtrade/plugins/pairlist/IPairList.py
--- a/freqtrade/plugins/pairlist/IPairList.py
+++ b/freqtrade/plugins/pairlist/IPairList.py
@@ -168,7 +168,7 @@ def _whitelist_for_active_markets(self, pairlist: List[str]) -> List[str]:
             # Check if market is active
             market = markets[pair]
             if not market_is_active(market):
-                logger.info(f"Ignoring {pair} from whitelist. Market is not active.")
+                self.log_once(f"Ignoring {pair} from whitelist. Market is not active.", logger.info)
                 continue
             if pair not in sanitized_whitelist:
                 sanitized_whitelist.append(pair)
diff --git a/freqtrade/plugins/pairlistmanager.py b/freqtrade/plugins/pairlistmanager.py
--- a/freqtrade/plugins/pairlistmanager.py
+++ b/freqtrade/plugins/pairlistmanager.py
@@ -59,17 +59,6 @@ def expanded_blacklist(self) -> List[str]:
         """The expanded blacklist (including wildcard expansion)"""
         return expand_pairlist(self._blacklist, self._exchange.get_markets().keys())
 
-    @property
-    def expanded_whitelist_keep_invalid(self) -> List[str]:
-        """The expanded whitelist (including wildcard expansion), maintaining invalid pairs"""
-        return expand_pairlist(self._whitelist, self._exchange.get_markets().keys(),
-                               keep_invalid=True)
-
-    @property
-    def expanded_whitelist(self) -> List[str]:
-        """The expanded whitelist (including wildcard expansion), filtering invalid pairs"""
-        return expand_pairlist(self._whitelist, self._exchange.get_markets().keys())
-
     @property
     def name_list(self) -> List[str]:
         """Get list of loaded Pairlist Handler names"""
@@ -153,10 +142,8 @@ def verify_whitelist(self, pairlist: List[str], logmethod,
         :return: pairlist - whitelisted pairs
         """
         try:
-            if keep_invalid:
-                whitelist = self.expanded_whitelist_keep_invalid
-            else:
-                whitelist = self.expanded_whitelist
+
+            whitelist = expand_pairlist(pairlist, self._exchange.get_markets().keys(), keep_invalid)
         except ValueError as err:
             logger.error(f"Pair whitelist contains an invalid Wildcard: {err}")
             return []


Test patch:
diff --git a/tests/plugins/test_pairlist.py b/tests/plugins/test_pairlist.py
--- a/tests/plugins/test_pairlist.py
+++ b/tests/plugins/test_pairlist.py
@@ -126,6 +126,20 @@ def test_load_pairlist_noexist(mocker, markets, default_conf):
                                        default_conf, {}, 1)
 
 
+def test_load_pairlist_verify_multi(mocker, markets, default_conf):
+    freqtrade = get_patched_freqtradebot(mocker, default_conf)
+    mocker.patch('freqtrade.exchange.Exchange.markets', PropertyMock(return_value=markets))
+    plm = PairListManager(freqtrade.exchange, default_conf)
+    # Call different versions one after the other, should always consider what was passed in
+    # and have no side-effects (therefore the same check multiple times)
+    assert plm.verify_whitelist(['ETH/BTC', 'XRP/BTC', ], print) == ['ETH/BTC', 'XRP/BTC']
+    assert plm.verify_whitelist(['ETH/BTC', 'XRP/BTC', 'BUUU/BTC'], print) == ['ETH/BTC', 'XRP/BTC']
+    assert plm.verify_whitelist(['XRP/BTC', 'BUUU/BTC'], print) == ['XRP/BTC']
+    assert plm.verify_whitelist(['ETH/BTC', 'XRP/BTC', ], print) == ['ETH/BTC', 'XRP/BTC']
+    assert plm.verify_whitelist(['ETH/USDT', 'XRP/USDT', ], print) == ['ETH/USDT', ]
+    assert plm.verify_whitelist(['ETH/BTC', 'XRP/BTC', ], print) == ['ETH/BTC', 'XRP/BTC']
+
+
 def test_refresh_market_pair_not_in_whitelist(mocker, markets, static_pl_conf):
 
     freqtrade = get_patched_freqtradebot(mocker, static_pl_conf)