Repository: freqtrade/freqtrade

Issue:
Crash on PrecisionFilter logic
Just got this crash from the PrecisionFilter on a develop based new branch, running it on FTX leveraged tokens. This error never happened before.

```
2020-03-28 00:43:49,891 - freqtrade.commands.trade_commands - INFO - worker found ... calling exit
2020-03-28 00:43:49,891 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': status, 'status': 'process died'}
2020-03-28 00:43:49,953 - freqtrade.freqtradebot - INFO - Cleaning up modules ...
2020-03-28 00:43:49,953 - freqtrade.rpc.rpc_manager - INFO - Cleaning up rpc modules ...
2020-03-28 00:44:05,419 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/commands/trade_commands.py", line 20, in start_trading
    worker.run()
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/worker.py", line 66, in run
    state = self._worker(old_state=state)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/worker.py", line 104, in _worker
    self._throttle(func=self._process_running, throttle_secs=self._throttle_secs)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/worker.py", line 125, in _throttle
    result = func(*args, **kwargs)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/worker.py", line 139, in _process_running
    self.freqtrade.process()
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/freqtradebot.py", line 142, in process
    self.active_pair_whitelist = self._refresh_whitelist(trades)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/freqtradebot.py", line 168, in _refresh_whitelist
    self.pairlists.refresh_pairlist()
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/pairlist/pairlistmanager.py", line 91, in refresh_pairlist
    pairlist = pl.filter_pairlist(pairlist, tickers)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/pairlist/PrecisionFilter.py", line 59, in filter_pairlist
    if not ticker or (stoploss and not self._validate_precision_filter(ticker, stoploss)):
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/pairlist/PrecisionFilter.py", line 36, in _validate_precision_filter
    stop_price = ticker['ask'] * stoploss
TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'
```
Crash on PrecisionFilter logic
Just got this crash from the PrecisionFilter on a develop based new branch, running it on FTX leveraged tokens. This error never happened before.

```
2020-03-28 00:43:49,891 - freqtrade.commands.trade_commands - INFO - worker found ... calling exit
2020-03-28 00:43:49,891 - freqtrade.rpc.rpc_manager - INFO - Sending rpc message: {'type': status, 'status': 'process died'}
2020-03-28 00:43:49,953 - freqtrade.freqtradebot - INFO - Cleaning up modules ...
2020-03-28 00:43:49,953 - freqtrade.rpc.rpc_manager - INFO - Cleaning up rpc modules ...
2020-03-28 00:44:05,419 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/commands/trade_commands.py", line 20, in start_trading
    worker.run()
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/worker.py", line 66, in run
    state = self._worker(old_state=state)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/worker.py", line 104, in _worker
    self._throttle(func=self._process_running, throttle_secs=self._throttle_secs)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/worker.py", line 125, in _throttle
    result = func(*args, **kwargs)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/worker.py", line 139, in _process_running
    self.freqtrade.process()
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/freqtradebot.py", line 142, in process
    self.active_pair_whitelist = self._refresh_whitelist(trades)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/freqtradebot.py", line 168, in _refresh_whitelist
    self.pairlists.refresh_pairlist()
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/pairlist/pairlistmanager.py", line 91, in refresh_pairlist
    pairlist = pl.filter_pairlist(pairlist, tickers)
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/pairlist/PrecisionFilter.py", line 59, in filter_pairlist
    if not ticker or (stoploss and not self._validate_precision_filter(ticker, stoploss)):
  File "/Users/yazeed/Sites/freqtradeLATEST/freqtrade/pairlist/PrecisionFilter.py", line 36, in _validate_precision_filter
    stop_price = ticker['ask'] * stoploss
TypeError: unsupported operand type(s) for *: 'NoneType' and 'float'
```


Discussion:
well yeah - FTX has the issue that they have pairs which don't have ANY trades for multiple days  - and also don't have orders on one side of the orderbook.

You won't want to trade these - you may be able to buy (you're the only buyer) - but it's impossible to sell (or viceversa) .... but it shouldn't crash the bot ...

I think i've fixed that for `get_*_rate()` a while ago (if you happen to have that pair in your static pairlist)  ... but didn't think of the precisionfilter ...
These tokens are not impossible to sell on FTX, there's always market makers providing liquidity. Remember that Leveraged tokens do not rely on orderbooks and can be redeemed or created for their NAV.

well yeah - FTX has the issue that they have pairs which don't have ANY trades for multiple days  - and also don't have orders on one side of the orderbook.

You won't want to trade these - you may be able to buy (you're the only buyer) - but it's impossible to sell (or viceversa) .... but it shouldn't crash the bot ...

I think i've fixed that for `get_*_rate()` a while ago (if you happen to have that pair in your static pairlist)  ... but didn't think of the precisionfilter ...
These tokens are not impossible to sell on FTX, there's always market makers providing liquidity. Remember that Leveraged tokens do not rely on orderbooks and can be redeemed or created for their NAV.


Resolving patch:
diff --git a/freqtrade/pairlist/PriceFilter.py b/freqtrade/pairlist/PriceFilter.py
--- a/freqtrade/pairlist/PriceFilter.py
+++ b/freqtrade/pairlist/PriceFilter.py
@@ -37,6 +37,12 @@ def _validate_ticker_lowprice(self, ticker) -> bool:
         :param ticker: ticker dict as returned from ccxt.load_markets()
         :return: True if the pair can stay, false if it should be removed
         """
+        if ticker['last'] is None:
+
+            self.log_on_refresh(logger.info,
+                                f"Removed {ticker['symbol']} from whitelist, because "
+                                "ticker['last'] is empty (Usually no trade in the last 24h).")
+            return False
         compare = ticker['last'] + self._exchange.price_get_one_pip(ticker['symbol'],
                                                                     ticker['last'])
         changeperc = (compare - ticker['last']) / ticker['last']
@@ -47,7 +53,6 @@ def _validate_ticker_lowprice(self, ticker) -> bool:
         return True
 
     def filter_pairlist(self, pairlist: List[str], tickers: Dict) -> List[str]:
-
         """
         Filters and sorts pairlist and returns the whitelist again.
         Called on each bot iteration - please use internal caching if necessary


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -739,6 +739,31 @@ def shitcoinmarkets(markets):
             "future": False,
             "active": True
         },
+        'ADAHALF/USDT': {
+            "percentage": True,
+            "tierBased": False,
+            "taker": 0.001,
+            "maker": 0.001,
+            "precision": {
+                "base": 8,
+                "quote": 8,
+                "amount": 2,
+                "price": 4
+            },
+            "limits": {
+            },
+            "id": "ADAHALFUSDT",
+            "symbol": "ADAHALF/USDT",
+            "base": "ADAHALF",
+            "quote": "USDT",
+            "baseId": "ADAHALF",
+            "quoteId": "USDT",
+            "info": {},
+            "type": "spot",
+            "spot": True,
+            "future": False,
+            "active": True
+    },
         })
     return shitmarkets
 
@@ -1243,6 +1268,29 @@ def tickers():
             "quoteVolume": 323652.075405,
             "info": {}
         },
+        # Example of leveraged pair with incomplete info
+        "ADAHALF/USDT": {
+            "symbol": "ADAHALF/USDT",
+            "timestamp": 1580469388244,
+            "datetime": "2020-01-31T11:16:28.244Z",
+            "high": None,
+            "low": None,
+            "bid": 0.7305,
+            "bidVolume": None,
+            "ask": 0.7342,
+            "askVolume": None,
+            "vwap": None,
+            "open": None,
+            "close": None,
+            "last": None,
+            "previousClose": None,
+            "change": None,
+            "percentage": 2.628,
+            "average": None,
+            "baseVolume": 0.0,
+            "quoteVolume": 0.0,
+            "info": {}
+        },
     })
 
 
diff --git a/tests/pairlist/test_pairlist.py b/tests/pairlist/test_pairlist.py
--- a/tests/pairlist/test_pairlist.py
+++ b/tests/pairlist/test_pairlist.py
@@ -163,7 +163,7 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf):
     ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "bidVolume"}],
         "BTC",  ['HOT/BTC', 'FUEL/BTC', 'XRP/BTC', 'LTC/BTC', 'TKN/BTC']),
     ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}],
-        "USDT", ['ETH/USDT', 'NANO/USDT']),
+        "USDT", ['ETH/USDT', 'NANO/USDT', 'ADAHALF/USDT']),
     # No pair for ETH ...
     ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"}],
      "ETH", []),
@@ -177,6 +177,10 @@ def test_VolumePairList_refresh_empty(mocker, markets_empty, whitelist_conf):
     ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"},
       {"method": "PriceFilter", "low_price_ratio": 0.03}],
         "BTC", ['ETH/BTC', 'TKN/BTC', 'LTC/BTC', 'XRP/BTC']),
+    # PriceFilter and VolumePairList
+    ([{"method": "VolumePairList", "number_assets": 5, "sort_key": "quoteVolume"},
+      {"method": "PriceFilter", "low_price_ratio": 0.03}],
+        "USDT", ['ETH/USDT', 'NANO/USDT']),
     # Hot is removed by precision_filter, Fuel by low_price_filter.
     ([{"method": "VolumePairList", "number_assets": 6, "sort_key": "quoteVolume"},
       {"method": "PrecisionFilter"},
@@ -221,7 +225,9 @@ def test_VolumePairList_whitelist_gen(mocker, whitelist_conf, shitcoinmarkets, t
             assert log_has_re(r'^Removed .* from whitelist, because stop price .* '
                               r'would be <= stop limit.*', caplog)
         if pairlist['method'] == 'PriceFilter':
-            assert log_has_re(r'^Removed .* from whitelist, because 1 unit is .*%$', caplog)
+            assert (log_has_re(r'^Removed .* from whitelist, because 1 unit is .*%$', caplog) or
+                    log_has_re(r"^Removed .* from whitelist, because ticker\['last'\] is empty.*",
+                               caplog))
 
 
 def test_gen_pair_whitelist_not_supported(mocker, default_conf, tickers) -> None: