Repository: freqtrade/freqtrade

Issue:
Backtesting: ValueError: max() arg is an empty sequence
(`Python 3.6.8 :: Anaconda, Inc`)
(`ccxt==1.18.667`)
Develop
Last Commit ID:`0d8b572a171e767e3429699c62250816294ede53`)
  

The most recent error in backtesting:

```
2019-06-19 11:07:12,301 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/user/Github/freqtrade/freqtrade/main.py", line 46, in main
    args.func(args)
  File "/home/user/Github/freqtrade/freqtrade/optimize/__init__.py", line 61, in start_backtesting
    backtesting.start()
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 449, in start
    'end_date': max_date,
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 378, in backtest
    max_open_trades)
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 254, in _get_sell_trade_entry
    sell_row.sell, low=sell_row.low, high=sell_row.high)
  File "/home/user/Github/freqtrade/freqtrade/strategy/interface.py", line 280, in should_sell
    if self.min_roi_reached(trade=trade, current_profit=current_profit, current_time=date):
  File "/home/user/Github/freqtrade/freqtrade/strategy/interface.py", line 359, in min_roi_reached
    roi_entry = max(list(filter(lambda x: trade_dur >= x, self.minimal_roi.keys())))
ValueError: max() arg is an empty sequence


```
Backtesting: ValueError: max() arg is an empty sequence
(`Python 3.6.8 :: Anaconda, Inc`)
(`ccxt==1.18.667`)
Develop
Last Commit ID:`0d8b572a171e767e3429699c62250816294ede53`)
  

The most recent error in backtesting:

```
2019-06-19 11:07:12,301 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/user/Github/freqtrade/freqtrade/main.py", line 46, in main
    args.func(args)
  File "/home/user/Github/freqtrade/freqtrade/optimize/__init__.py", line 61, in start_backtesting
    backtesting.start()
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 449, in start
    'end_date': max_date,
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 378, in backtest
    max_open_trades)
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 254, in _get_sell_trade_entry
    sell_row.sell, low=sell_row.low, high=sell_row.high)
  File "/home/user/Github/freqtrade/freqtrade/strategy/interface.py", line 280, in should_sell
    if self.min_roi_reached(trade=trade, current_profit=current_profit, current_time=date):
  File "/home/user/Github/freqtrade/freqtrade/strategy/interface.py", line 359, in min_roi_reached
    roi_entry = max(list(filter(lambda x: trade_dur >= x, self.minimal_roi.keys())))
ValueError: max() arg is an empty sequence


```


Discussion:
What's the value of the "minimal_roi" option set in your config and/or in the strategy?
What timeframe do you use for backtesting?

> What's the value of the "minimal_roi" option set in your config and/or in the strategy?
> What timeframe do you use for backtesting?

minimal_roi is using the values returned from the hyperopts and I am using 15m in this one. The config one is overriding the strategy.py file and the command line overrides ticker timeframe 
it could be that `t_1 ... t_2 ...` for "minimal_roi" is set quite high because I didn't change it from the 1d tests I tried, and it is pocketing the trailing stop returning 'no value' for "minimal_roi", but I don't know if/why that should be a problem
What is the exact command you're running to get this error?
python3 freqtrade -s StrategyOne backtesting -i 15m
how does StrategyOne define `minimal_roi`?

this error does only happen if nothing is in the dict ...
> how does StrategyOne define `minimal_roi`?
> 
> this error does only happen if nothing is in the dict ...

ok, it's left unchanged because by minimal_roi is in the config file
I had realised I missed a parameter, but now I get 

```
2019-06-19 15:39:28,457 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/user/Github/freqtrade/freqtrade/main.py", line 46, in main
    args.func(args)
  File "/home/user/Github/freqtrade/freqtrade/optimize/__init__.py", line 61, in start_backtesting
    backtesting.start()
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 449, in start
    'end_date': max_date,
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 335, in backtest
    ticker: Dict = self._get_ticker_list(processed)
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 220, in _get_ticker_list
    self.advise_buy(pair_data, {'pair': pair}), {'pair': pair})[headers].copy()
  File "/home/user/Github/freqtrade/freqtrade/strategy/interface.py", line 401, in advise_buy
    return self.populate_buy_trend(dataframe, metadata)
  File "/home/user/Github/freqtrade/user_data/strategies/test_strat_to.py", line 288, in populate_buy_trend
    'buy'] = 1
  File "/home/user/Github/freqtrade/freqtrade/vendor/qtpylib/indicators.py", line 244, in crossed_above
    return crossed(series1, series2, "above")
  File "/home/user/Github/freqtrade/freqtrade/vendor/qtpylib/indicators.py", line 230, in crossed
    above = pd.Series((series1 > series2) & (
  File "/home/user/.conda/envs/freqtrade/lib/python3.6/site-packages/pandas/core/ops.py", line 1745, in wrapper
    raise ValueError('Lengths must match to compare')
ValueError: Lengths must match to compare
```
i had earlier gotten that same mistake, it could be that the pairlist includes newly added currencies and return the one with 'expected x frames, got x_less frames'.
gonna check if removing them from the list would do anything or something to do with qtpylib
The qtpylib error was solved, a minor issue. But the roi error remains there, I tried with the config.example.json one and it is the same error.
> > how does StrategyOne define `minimal_roi`?
> > this error does only happen if nothing is in the dict ...
> 
> ok, it's left unchanged because by minimal_roi is in the config file

Okay, how does config define `minimal_roi`?

it seemed to work with hyperopts, once deleted it works but only from the .py file so it was not overriding
So the problem is solved, right?

yeah, but only by removing minimal_roi from config
"minimal_roi" : {
"x" : xx 
"x" : xx
"x" : xx
"x" : xx
} 
@xmatthias anyway, a check should be added here, avoiding exception which came from max().

@gitgitgat could you post the exact value of the minimal_roi you used?
actually the error is being raised again, it only works if minimal roi uses the default parameters
What's the value of the "minimal_roi" option set in your config and/or in the strategy?
What timeframe do you use for backtesting?

> What's the value of the "minimal_roi" option set in your config and/or in the strategy?
> What timeframe do you use for backtesting?

minimal_roi is using the values returned from the hyperopts and I am using 15m in this one. The config one is overriding the strategy.py file and the command line overrides ticker timeframe 
it could be that `t_1 ... t_2 ...` for "minimal_roi" is set quite high because I didn't change it from the 1d tests I tried, and it is pocketing the trailing stop returning 'no value' for "minimal_roi", but I don't know if/why that should be a problem
What is the exact command you're running to get this error?
python3 freqtrade -s StrategyOne backtesting -i 15m
how does StrategyOne define `minimal_roi`?

this error does only happen if nothing is in the dict ...
> how does StrategyOne define `minimal_roi`?
> 
> this error does only happen if nothing is in the dict ...

ok, it's left unchanged because by minimal_roi is in the config file
I had realised I missed a parameter, but now I get 

```
2019-06-19 15:39:28,457 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/user/Github/freqtrade/freqtrade/main.py", line 46, in main
    args.func(args)
  File "/home/user/Github/freqtrade/freqtrade/optimize/__init__.py", line 61, in start_backtesting
    backtesting.start()
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 449, in start
    'end_date': max_date,
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 335, in backtest
    ticker: Dict = self._get_ticker_list(processed)
  File "/home/user/Github/freqtrade/freqtrade/optimize/backtesting.py", line 220, in _get_ticker_list
    self.advise_buy(pair_data, {'pair': pair}), {'pair': pair})[headers].copy()
  File "/home/user/Github/freqtrade/freqtrade/strategy/interface.py", line 401, in advise_buy
    return self.populate_buy_trend(dataframe, metadata)
  File "/home/user/Github/freqtrade/user_data/strategies/test_strat_to.py", line 288, in populate_buy_trend
    'buy'] = 1
  File "/home/user/Github/freqtrade/freqtrade/vendor/qtpylib/indicators.py", line 244, in crossed_above
    return crossed(series1, series2, "above")
  File "/home/user/Github/freqtrade/freqtrade/vendor/qtpylib/indicators.py", line 230, in crossed
    above = pd.Series((series1 > series2) & (
  File "/home/user/.conda/envs/freqtrade/lib/python3.6/site-packages/pandas/core/ops.py", line 1745, in wrapper
    raise ValueError('Lengths must match to compare')
ValueError: Lengths must match to compare
```
i had earlier gotten that same mistake, it could be that the pairlist includes newly added currencies and return the one with 'expected x frames, got x_less frames'.
gonna check if removing them from the list would do anything or something to do with qtpylib
The qtpylib error was solved, a minor issue. But the roi error remains there, I tried with the config.example.json one and it is the same error.
> > how does StrategyOne define `minimal_roi`?
> > this error does only happen if nothing is in the dict ...
> 
> ok, it's left unchanged because by minimal_roi is in the config file

Okay, how does config define `minimal_roi`?

it seemed to work with hyperopts, once deleted it works but only from the .py file so it was not overriding
So the problem is solved, right?

yeah, but only by removing minimal_roi from config
"minimal_roi" : {
"x" : xx 
"x" : xx
"x" : xx
"x" : xx
} 
@xmatthias anyway, a check should be added here, avoiding exception which came from max().

@gitgitgat could you post the exact value of the minimal_roi you used?
actually the error is being raised again, it only works if minimal roi uses the default parameters

Resolving patch:
diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -253,22 +253,20 @@ def _get_sell_trade_entry(
             sell = self.strategy.should_sell(trade, sell_row.open, sell_row.date, sell_row.buy,
                                              sell_row.sell, low=sell_row.low, high=sell_row.high)
             if sell.sell_flag:
-
                 trade_dur = int((sell_row.date - buy_row.date).total_seconds() // 60)
                 # Special handling if high or low hit STOP_LOSS or ROI
                 if sell.sell_type in (SellType.STOP_LOSS, SellType.TRAILING_STOP_LOSS):
                     # Set close_rate to stoploss
                     closerate = trade.stop_loss
                 elif sell.sell_type == (SellType.ROI):
-                    # get next entry in min_roi > to trade duration
-                    # Interface.py skips on trade_duration <= duration
-                    roi_entry = max(list(filter(lambda x: trade_dur >= x,
-                                                self.strategy.minimal_roi.keys())))
-                    roi = self.strategy.minimal_roi[roi_entry]
-
-                    # - (Expected abs profit + open_rate + open_fee) / (fee_close -1)
-                    closerate = - (trade.open_rate * roi + trade.open_rate *
-                                   (1 + trade.fee_open)) / (trade.fee_close - 1)
+                    roi = self.strategy.min_roi_reached_entry(trade_dur)
+                    if roi is not None:
+                        # - (Expected abs profit + open_rate + open_fee) / (fee_close -1)
+                        closerate = - (trade.open_rate * roi + trade.open_rate *
+                                       (1 + trade.fee_open)) / (trade.fee_close - 1)
+                    else:
+                        # This should not be reached...
+                        closerate = sell_row.open
                 else:
                     closerate = sell_row.open
 
diff --git a/freqtrade/strategy/interface.py b/freqtrade/strategy/interface.py
--- a/freqtrade/strategy/interface.py
+++ b/freqtrade/strategy/interface.py
@@ -6,7 +6,7 @@
 from abc import ABC, abstractmethod
 from datetime import datetime
 from enum import Enum
-from typing import Dict, List, NamedTuple, Tuple
+from typing import Dict, List, NamedTuple, Optional, Tuple
 import warnings
 
 import arrow
@@ -347,23 +347,32 @@ def stop_loss_reached(self, current_rate: float, trade: Trade,
 
         return SellCheckTuple(sell_flag=False, sell_type=SellType.NONE)
 
+    def min_roi_reached_entry(self, trade_dur: int) -> Optional[float]:
+        """
+        Based on trade duration defines the ROI entry that may have been reached.
+        :param trade_dur: trade duration in minutes
+        :return: minimal ROI entry value or None if none proper ROI entry was found.
+        """
+        # Get highest entry in ROI dict where key <= trade-duration
+        roi_list = list(filter(lambda x: x <= trade_dur, self.minimal_roi.keys()))
+        if not roi_list:
+            return None
+        roi_entry = max(roi_list)
+        return self.minimal_roi[roi_entry]
+
     def min_roi_reached(self, trade: Trade, current_profit: float, current_time: datetime) -> bool:
         """
-        Based an earlier trade and current price and ROI configuration, decides whether bot should
+        Based on trade duration, current price and ROI configuration, decides whether bot should
         sell. Requires current_profit to be in percent!!
-        :return True if bot should sell at current rate
+        :return: True if bot should sell at current rate
         """
-
         # Check if time matches and current rate is above threshold
-        trade_dur = (current_time.timestamp() - trade.open_date.timestamp()) / 60
-
-        # Get highest entry in ROI dict where key >= trade-duration
-        roi_entry = max(list(filter(lambda x: trade_dur >= x, self.minimal_roi.keys())))
-        threshold = self.minimal_roi[roi_entry]
-        if current_profit > threshold:
-            return True
-
-        return False
+        trade_dur = int((current_time.timestamp() - trade.open_date.timestamp()) // 60)
+        roi = self.min_roi_reached_entry(trade_dur)
+        if roi is None:
+            return False
+        else:
+            return current_profit > roi
 
     def tickerdata_to_dataframe(self, tickerdata: Dict[str, List]) -> Dict[str, DataFrame]:
         """


Test patch:
diff --git a/freqtrade/tests/strategy/test_interface.py b/freqtrade/tests/strategy/test_interface.py
--- a/freqtrade/tests/strategy/test_interface.py
+++ b/freqtrade/tests/strategy/test_interface.py
@@ -185,6 +185,39 @@ def test_min_roi_reached2(default_conf, fee) -> None:
         assert strategy.min_roi_reached(trade, 0.31, arrow.utcnow().shift(minutes=-2).datetime)
 
 
+def test_min_roi_reached3(default_conf, fee) -> None:
+
+    # test for issue #1948
+    min_roi = {20: 0.07,
+               30: 0.05,
+               55: 0.30,
+               }
+    strategy = DefaultStrategy(default_conf)
+    strategy.minimal_roi = min_roi
+    trade = Trade(
+            pair='ETH/BTC',
+            stake_amount=0.001,
+            open_date=arrow.utcnow().shift(hours=-1).datetime,
+            fee_open=fee.return_value,
+            fee_close=fee.return_value,
+            exchange='bittrex',
+            open_rate=1,
+    )
+
+    assert not strategy.min_roi_reached(trade, 0.02, arrow.utcnow().shift(minutes=-56).datetime)
+    assert not strategy.min_roi_reached(trade, 0.12, arrow.utcnow().shift(minutes=-56).datetime)
+
+    assert not strategy.min_roi_reached(trade, 0.04, arrow.utcnow().shift(minutes=-39).datetime)
+    assert strategy.min_roi_reached(trade, 0.071, arrow.utcnow().shift(minutes=-39).datetime)
+
+    assert not strategy.min_roi_reached(trade, 0.04, arrow.utcnow().shift(minutes=-26).datetime)
+    assert strategy.min_roi_reached(trade, 0.06, arrow.utcnow().shift(minutes=-26).datetime)
+
+    # Should not trigger with 20% profit since after 55 minutes only 30% is active.
+    assert not strategy.min_roi_reached(trade, 0.20, arrow.utcnow().shift(minutes=-2).datetime)
+    assert strategy.min_roi_reached(trade, 0.31, arrow.utcnow().shift(minutes=-2).datetime)
+
+
 def test_analyze_ticker_default(ticker_history, mocker, caplog) -> None:
     caplog.set_level(logging.DEBUG)
     ind_mock = MagicMock(side_effect=lambda x, meta: x)