Repository: freqtrade/freqtrade

Issue:
Error using okex - strategy requires 480 candles to start
<!-- 
Have you searched for similar issues before posting it?

If you have discovered a bug in the bot, please [search the issue tracker](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue). 
If it hasn't been reported, please create a new issue.

Please do not use bug reports to request new features.
-->

## Describe your environment

  * Operating system: Ubuntu
  * Python Version: Python 3.8.10
  * CCXT version: 1.60.11:
  * Freqtrade Version:  develop-a1e887803
  
## Describe the problem:

*Explain the problem you have encountered*

### Steps to reproduce:

  1. Just config OKEX exchange and error apears
  
### Observed Results:

Crash freqtrade.
  * What did you expect to happen?

### Relevant code exceptions or logs

Note: Please copy/paste text of the messages, no screenshots of logs please.

  ```
  2021-11-03 16:52:41,167 - freqtrade.exchange.exchange - INFO - Using CCXT 1.60.11
2021-11-03 16:52:41,168 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True}
2021-11-03 16:52:41,171 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True, 'rateLimit': 105}
2021-11-03 16:52:41,174 - freqtrade.exchange.exchange - INFO - Using Exchange "OKEX"
2021-11-03 16:52:44,754 - freqtrade.commands.trade_commands - ERROR - This strategy requires 480 candles to start. OKEX only provides 95 for 5m.
2021-11-03 16:52:44,754 - freqtrade.commands.trade_commands - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/lds/freqtradev2/freqtrade/commands/trade_commands.py", line 18, in start_trading
    worker = Worker(args)
  File "/home/lds/freqtradev2/freqtrade/worker.py", line 35, in __init__
    self._init(False)
  File "/home/lds/freqtradev2/freqtrade/worker.py", line 52, in _init
    self.freqtrade = FreqtradeBot(self._config)
  File "/home/lds/freqtradev2/freqtrade/freqtradebot.py", line 65, in __init__
    self.exchange = ExchangeResolver.load_exchange(self.config['exchange']['name'], self.config)
  File "/home/lds/freqtradev2/freqtrade/resolvers/exchange_resolver.py", line 32, in load_exchange
    exchange = ExchangeResolver._load_exchange(exchange_name,
  File "/home/lds/freqtradev2/freqtrade/resolvers/exchange_resolver.py", line 54, in _load_exchange
    exchange = ex_class(**kwargs)
  File "/home/lds/freqtradev2/freqtrade/exchange/exchange.py", line 158, in __init__
    self.validate_required_startup_candles(config.get('startup_candle_count', 0),
  File "/home/lds/freqtradev2/freqtrade/exchange/exchange.py", line 481, in validate_required_startup_candles
    raise OperationalException(
freqtrade.exceptions.OperationalException: This strategy requires 480 candles to start. OKEX only provides 95 for 5m.

  ```

startup candle count beyond 992 ? 
<!-- 
Have you searched for similar issues before posting it?

Please only post bug-reports related to strategies in this repository.
For bugs with Freqtrade, please use the [freqtrade issue tracker](https://github.com/freqtrade/freqtrade/issues)

Please do not use bug reports to request new features.
-->

## Describe your environment

  * Operating system: ____
  * Python Version: _____ (`python -V`)
  * CCXT version: _____ (`pip freeze | grep ccxt`)
  * Freqtrade Version: ____ (`freqtrade -V` or `docker-compose run --rm freqtrade -V` for Freqtrade running in docker)
  * Strategy: _____
  
Note: All bug reports will be closed without further comment if the above template is deleted or not filled out.

## Describe the problem:

hello, 
I have a  strategy on 1m timeframe but I need to increase the number of entry point to increase the number of startup_candle_count so far I can only go max to 992 do you know how I could increase to 1440 ? So I can get 24h data on 1m timeframe as a ref ? 

### Steps to reproduce:

1. put timeframe to '1m'
2. put startup_candle_count
3. run the bot with the command sudo docker-compose up -d 

  
### Observed Results:

Not able to go beyond 992 startup_candle_count

### Relevant code exceptions or logs

Note: Please copy/paste text of the messages, no screenshots of logs please.

    # Number of candles the strategy requires before producing valid signals
    startup_candle_count: int = 992


Discussion:
I think the exception is quite clear ... Your strategy requires historic 480 candles - but okex only provides 95 (which is technically not true, ti'll be 98, but we apply a safety cushion to the value, as sometimes candles can be missing).

Quite simply - have your strategy require less historic data to trade with it on okex.

I'm working on improving this (specifically for/because of OKEX) - but it's not ready yet - so at the moment, this strategy (assuming the history requirement is correct) will not work.

Hi xmatthias, many thanks for ur reply...

In my test, this strategy works in freqtrade develop-20a61e03d but no in develop-a1e887803.


older versions of freqtrade may start with this strategy in combination with okex - however but it'll not work correctly, as okex only provides 100 candles of history (this didn't change...).
So the strategy starts - but indicators with a lookback further than 100 will not calculate correctly, probably causing no buys (or at least no buys with this particular signal).
you'll also not have been able to backtest against OKEX, as data-download did not work correctly (I'd always recommend backtesting against the exchange / markets you're really working against).

So it's not that it works correctly - it just doesn't alert you that something is wrong.
for backtesting, also values of more than 992 should be possible - but without having a reason to do it that way (see below).

For real mode (trade, dry-run) - you'll be limited to "one ohlcv request"  - which will be different per exchange.
It'll be roughly ~1000 for binance (fewer if there was a downtime) - 1400 for FTX, and ~500 for bittrex.

If you need more than the available data, then [informative pairs](https://www.freqtrade.io/en/stable/strategy-customization/#additional-data-informative_pairs) on longer timeframes is usually the solution - as i doubt that you'll need the indicators on 1m data for that long of a period.


ok thanks the response is clear. I'll try to explore informative pairs in that case. 

Resolving patch:
diff --git a/freqtrade/exchange/binance.py b/freqtrade/exchange/binance.py
--- a/freqtrade/exchange/binance.py
+++ b/freqtrade/exchange/binance.py
@@ -1,6 +1,6 @@
 """ Binance exchange subclass """
 import logging
-from typing import Dict, List
+from typing import Dict, List, Tuple
 
 import arrow
 import ccxt
@@ -93,8 +93,9 @@ def stoploss(self, pair: str, amount: float, stop_price: float, order_types: Dic
             raise OperationalException(e) from e
 
     async def _async_get_historic_ohlcv(self, pair: str, timeframe: str,
-                                        since_ms: int, is_new_pair: bool
-                                        ) -> List:
+                                        since_ms: int, is_new_pair: bool = False,
+                                        raise_: bool = False
+                                        ) -> Tuple[str, str, List]:
         """
         Overwrite to introduce "fast new pair" functionality by detecting the pair's listing date
         Does not work for other exchanges, which don't return the earliest data when called with "0"
@@ -107,4 +108,5 @@ async def _async_get_historic_ohlcv(self, pair: str, timeframe: str,
                 logger.info(f"Candle-data for {pair} available starting with "
                             f"{arrow.get(since_ms // 1000).isoformat()}.")
         return await super()._async_get_historic_ohlcv(
-            pair=pair, timeframe=timeframe, since_ms=since_ms, is_new_pair=is_new_pair)
+            pair=pair, timeframe=timeframe, since_ms=since_ms, is_new_pair=is_new_pair,
+            raise_=raise_)
diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py
--- a/freqtrade/exchange/exchange.py
+++ b/freqtrade/exchange/exchange.py
@@ -7,7 +7,7 @@
 import inspect
 import logging
 from copy import deepcopy
-from datetime import datetime, timezone
+from datetime import datetime, timedelta, timezone
 from math import ceil
 from typing import Any, Dict, List, Optional, Tuple
 
@@ -155,8 +155,8 @@ def __init__(self, config: Dict[str, Any], validate: bool = True) -> None:
                 self.validate_pairs(config['exchange']['pair_whitelist'])
             self.validate_ordertypes(config.get('order_types', {}))
             self.validate_order_time_in_force(config.get('order_time_in_force', {}))
-            self.validate_required_startup_candles(config.get('startup_candle_count', 0),
-                                                   config.get('timeframe', ''))
+            self.required_candle_call_count = self.validate_required_startup_candles(
+                config.get('startup_candle_count', 0), config.get('timeframe', ''))
 
         # Converts the interval provided in minutes in config to seconds
         self.markets_refresh_interval: int = exchange_config.get(
@@ -471,16 +471,29 @@ def validate_order_time_in_force(self, order_time_in_force: Dict) -> None:
             raise OperationalException(
                 f'Time in force policies are not supported for {self.name} yet.')
 
-    def validate_required_startup_candles(self, startup_candles: int, timeframe: str) -> None:
+    def validate_required_startup_candles(self, startup_candles: int, timeframe: str) -> int:
         """
         Checks if required startup_candles is more than ohlcv_candle_limit().
         Requires a grace-period of 5 candles - so a startup-period up to 494 is allowed by default.
         """
         candle_limit = self.ohlcv_candle_limit(timeframe)
-        if startup_candles + 5 > candle_limit:
+        # Require one more candle - to account for the still open candle.
+        candle_count = startup_candles + 1
+        # Allow 5 calls to the exchange per pair
+        required_candle_call_count = int(
+            (candle_count / candle_limit) + (0 if candle_count % candle_limit == 0 else 1))
+
+        if required_candle_call_count > 5:
+            # Only allow 5 calls per pair to somewhat limit the impact
             raise OperationalException(
-                f"This strategy requires {startup_candles} candles to start. "
-                f"{self.name} only provides {candle_limit - 5} for {timeframe}.")
+                f"This strategy requires {startup_candles} candles to start, which is more than 5x "
+                f"the amount of candles {self.name} provides for {timeframe}.")
+
+        if required_candle_call_count > 1:
+            logger.warning(f"Using {required_candle_call_count} calls to get OHLCV. "
+                           f"This can result in slower operations for the bot. Please check "
+                           f"if you really need {startup_candles} candles for your strategy")
+        return required_candle_call_count
 
     def exchange_has(self, endpoint: str) -> bool:
         """
@@ -1205,9 +1218,11 @@ def get_historic_ohlcv(self, pair: str, timeframe: str,
         :param since_ms: Timestamp in milliseconds to get history from
         :return: List with candle (OHLCV) data
         """
-        return asyncio.get_event_loop().run_until_complete(
+        pair, timeframe, data = asyncio.get_event_loop().run_until_complete(
             self._async_get_historic_ohlcv(pair=pair, timeframe=timeframe,
                                            since_ms=since_ms, is_new_pair=is_new_pair))
+        logger.info(f"Downloaded data for {pair} with length {len(data)}.")
+        return data
 
     def get_historic_ohlcv_as_df(self, pair: str, timeframe: str,
                                  since_ms: int) -> DataFrame:
@@ -1223,8 +1238,9 @@ def get_historic_ohlcv_as_df(self, pair: str, timeframe: str,
                                   drop_incomplete=self._ohlcv_partial_candle)
 
     async def _async_get_historic_ohlcv(self, pair: str, timeframe: str,
-                                        since_ms: int, is_new_pair: bool
-                                        ) -> List:
+                                        since_ms: int, is_new_pair: bool = False,
+                                        raise_: bool = False
+                                        ) -> Tuple[str, str, List]:
         """
         Download historic ohlcv
         :param is_new_pair: used by binance subclass to allow "fast" new pair downloading
@@ -1248,15 +1264,17 @@ async def _async_get_historic_ohlcv(self, pair: str, timeframe: str,
             for res in results:
                 if isinstance(res, Exception):
                     logger.warning("Async code raised an exception: %s", res.__class__.__name__)
+                    if raise_:
+                        raise
                     continue
-                # Deconstruct tuple if it's not an exception
-                p, _, new_data = res
-                if p == pair:
-                    data.extend(new_data)
+                else:
+                    # Deconstruct tuple if it's not an exception
+                    p, _, new_data = res
+                    if p == pair:
+                        data.extend(new_data)
         # Sort data again after extending the result - above calls return in "async order"
         data = sorted(data, key=lambda x: x[0])
-        logger.info(f"Downloaded data for {pair} with length {len(data)}.")
-        return data
+        return pair, timeframe, data
 
     def refresh_latest_ohlcv(self, pair_list: ListPairsWithTimeframes, *,
                              since_ms: Optional[int] = None, cache: bool = True
@@ -1276,10 +1294,22 @@ def refresh_latest_ohlcv(self, pair_list: ListPairsWithTimeframes, *,
         cached_pairs = []
         # Gather coroutines to run
         for pair, timeframe in set(pair_list):
-            if (((pair, timeframe) not in self._klines)
+            if ((pair, timeframe) not in self._klines
                     or self._now_is_time_to_refresh(pair, timeframe)):
-                input_coroutines.append(self._async_get_candle_history(pair, timeframe,
-                                                                       since_ms=since_ms))
+                if not since_ms and self.required_candle_call_count > 1:
+                    # Multiple calls for one pair - to get more history
+                    one_call = timeframe_to_msecs(timeframe) * self.ohlcv_candle_limit(timeframe)
+                    move_to = one_call * self.required_candle_call_count
+                    now = timeframe_to_next_date(timeframe)
+                    since_ms = int((now - timedelta(seconds=move_to // 1000)).timestamp() * 1000)
+
+                if since_ms:
+                    input_coroutines.append(self._async_get_historic_ohlcv(
+                        pair, timeframe, since_ms=since_ms, raise_=True))
+                else:
+                    # One call ... "regular" refresh
+                    input_coroutines.append(self._async_get_candle_history(
+                        pair, timeframe, since_ms=since_ms))
             else:
                 logger.debug(
                     "Using cached candle (OHLCV) data for pair %s, timeframe %s ...",


Test patch:
diff --git a/tests/exchange/test_binance.py b/tests/exchange/test_binance.py
--- a/tests/exchange/test_binance.py
+++ b/tests/exchange/test_binance.py
@@ -126,13 +126,16 @@ async def test__async_get_historic_ohlcv_binance(default_conf, mocker, caplog):
     exchange._api_async.fetch_ohlcv = get_mock_coro(ohlcv)
 
     pair = 'ETH/BTC'
-    res = await exchange._async_get_historic_ohlcv(pair, "5m",
-                                                   1500000000000, is_new_pair=False)
+    respair, restf, res = await exchange._async_get_historic_ohlcv(
+        pair, "5m", 1500000000000, is_new_pair=False)
+    assert respair == pair
+    assert restf == '5m'
     # Call with very old timestamp - causes tons of requests
     assert exchange._api_async.fetch_ohlcv.call_count > 400
     # assert res == ohlcv
     exchange._api_async.fetch_ohlcv.reset_mock()
-    res = await exchange._async_get_historic_ohlcv(pair, "5m", 1500000000000, is_new_pair=True)
+    _, _, res = await exchange._async_get_historic_ohlcv(
+        pair, "5m", 1500000000000, is_new_pair=True)
 
     # Called twice - one "init" call - and one to get the actual data.
     assert exchange._api_async.fetch_ohlcv.call_count == 2
diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py
--- a/tests/exchange/test_exchange.py
+++ b/tests/exchange/test_exchange.py
@@ -941,9 +941,22 @@ def test_validate_required_startup_candles(default_conf, mocker, caplog):
     default_conf['startup_candle_count'] = 20
     ex = Exchange(default_conf)
     assert ex
-    default_conf['startup_candle_count'] = 600
-
-    with pytest.raises(OperationalException, match=r'This strategy requires 600.*'):
+    # assumption is that the exchange provides 500 candles per call.s
+    assert ex.validate_required_startup_candles(200, '5m') == 1
+    assert ex.validate_required_startup_candles(499, '5m') == 1
+    assert ex.validate_required_startup_candles(600, '5m') == 2
+    assert ex.validate_required_startup_candles(501, '5m') == 2
+    assert ex.validate_required_startup_candles(499, '5m') == 1
+    assert ex.validate_required_startup_candles(1000, '5m') == 3
+    assert ex.validate_required_startup_candles(2499, '5m') == 5
+    assert log_has_re(r'Using 5 calls to get OHLCV. This.*', caplog)
+
+    with pytest.raises(OperationalException, match=r'This strategy requires 2500.*'):
+        ex.validate_required_startup_candles(2500, '5m')
+
+    # Ensure the same also happens on init
+    default_conf['startup_candle_count'] = 6000
+    with pytest.raises(OperationalException, match=r'This strategy requires 6000.*'):
         Exchange(default_conf)
 
 
@@ -1506,6 +1519,7 @@ async def mock_candle_hist(pair, timeframe, since_ms):
     assert exchange._async_get_candle_history.call_count == 2
     # Returns twice the above OHLCV data
     assert len(ret) == 2
+    assert log_has_re(r'Downloaded data for .* with length .*\.', caplog)
 
     caplog.clear()
 
@@ -1587,12 +1601,13 @@ async def test__async_get_historic_ohlcv(default_conf, mocker, caplog, exchange_
     exchange._api_async.fetch_ohlcv = get_mock_coro(ohlcv)
 
     pair = 'ETH/USDT'
-    res = await exchange._async_get_historic_ohlcv(pair, "5m",
-                                                   1500000000000, is_new_pair=False)
+    respair, restf, res = await exchange._async_get_historic_ohlcv(
+        pair, "5m", 1500000000000, is_new_pair=False)
+    assert respair == pair
+    assert restf == '5m'
     # Call with very old timestamp - causes tons of requests
     assert exchange._api_async.fetch_ohlcv.call_count > 200
     assert res[0] == ohlcv[0]
-    assert log_has_re(r'Downloaded data for .* with length .*\.', caplog)
 
 
 def test_refresh_latest_ohlcv(mocker, default_conf, caplog) -> None:
@@ -1630,12 +1645,14 @@ def test_refresh_latest_ohlcv(mocker, default_conf, caplog) -> None:
     assert exchange._api_async.fetch_ohlcv.call_count == 2
     exchange._api_async.fetch_ohlcv.reset_mock()
 
+    exchange.required_candle_call_count = 2
     res = exchange.refresh_latest_ohlcv(pairs)
     assert len(res) == len(pairs)
 
     assert log_has(f'Refreshing candle (OHLCV) data for {len(pairs)} pairs', caplog)
     assert exchange._klines
-    assert exchange._api_async.fetch_ohlcv.call_count == 2
+    assert exchange._api_async.fetch_ohlcv.call_count == 4
+    exchange._api_async.fetch_ohlcv.reset_mock()
     for pair in pairs:
         assert isinstance(exchange.klines(pair), DataFrame)
         assert len(exchange.klines(pair)) > 0
@@ -1651,7 +1668,7 @@ def test_refresh_latest_ohlcv(mocker, default_conf, caplog) -> None:
     res = exchange.refresh_latest_ohlcv([('IOTA/ETH', '5m'), ('XRP/ETH', '5m')])
     assert len(res) == len(pairs)
 
-    assert exchange._api_async.fetch_ohlcv.call_count == 2
+    assert exchange._api_async.fetch_ohlcv.call_count == 0
     assert log_has(f"Using cached candle (OHLCV) data for pair {pairs[0][0]}, "
                    f"timeframe {pairs[0][1]} ...",
                    caplog)