Repository: freqtrade/freqtrade

Issue:
download-data 1m tickers produces empty files
## Describe your environment

  * Operating system: Ubuntu 20.04.3 LTS
  * Python Version:  Python 3.8.10
  * CCXT version: ccxt==1.55.83
  * Freqtrade Version: freqtrade develop-10d0987f4
    
Note: All issues other than enhancement requests will be closed without further comment if the above template is deleted or not filled out.

## Describe the problem:

*Explain the problem you have encountered*

When downloading 1m tickers from binance, I'm frequently getting empty json files, like `[]`. tickers other than 1m are working fine. 5m, 30m, 1h.

Sometimes I do get data with tickers, sometimes I don't.

I've tried increasing `rateLimit`.

### Steps to reproduce:

  1. freqtrade download-data --pairs- BTC/USDT -t 1m -c config.json

### Observed Results:

  * What happened?

When downloading for many pairs, data get's downloaded only for some of the pairs.

  * What did you expect to happen?

**Either data get's downloaded, or more meaningful errors thrown, to explain why data can't be downloaded. Or information about gaps, or statistics on the missing data.**

### Relevant code exceptions or logs

Along with this, I'm getting this errors
```
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,687 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-05 11:15:48,787 - freqtrade.exchange.exchange - INFO - Downloaded data for BTC/USDT with length 0.

```


Discussion:
1. Is there a way to increase retries? 
2. What download will do, if it can't download specific ticker after N retries ?
@xmatthias 

I can consitentently reproduce error with this command

```
freqtrade download-data --pairs ZEN/USDT -t 1m -c config.json --days 1345 -vvv
```
gives me
```
2021-09-08 20:45:03,876 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-08 20:45:03,876 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-08 20:45:03,876 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-08 20:45:03,876 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-08 20:45:03,876 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-08 20:45:03,876 - freqtrade.exchange.exchange - WARNING - Async code raised an exception: RuntimeError
2021-09-08 20:45:03,876 - freqtrade.exchange.exchange - INFO - Downloaded data for ZEN/USDT with length 0.
2021-09-08 20:45:03,880 - freqtrade.data.converter - DEBUG - Converting candle (OHLCV) data to dataframe for pair ZEN/USDT.
```

but this works fine
```
freqtrade download-data --pairs ZEN/USDT -t 1m -c config.json --days 10 -vvv
```
After some debugging, this is the error I'm getting from ccxt
```
throttle queue is over maxCapacity
```

https://github.com/ccxt/ccxt/blob/351bad11c010abbad977c9dbfad76a6fccaeaccf/python/ccxt/async_support/base/throttler.py#L17
looks like a good find - a fix will however probably take a bit  - as i think this is quite an edgecase - and as such low priority.

with the current settings, you can get back ~694 days (1M minutes) - if you have this much data of 1m pairs (for a decent amount of pairs) - it'll probably become a disk-problem as well (although I'd have to do the math for that).

@Potato334343 have the same problem ;-) you can if you want 1 minute candle from binance get them from https://github.com/binance/binance-public-data maybe little works to merge and parse. GL.

Resolving patch:
diff --git a/freqtrade/exchange/exchange.py b/freqtrade/exchange/exchange.py
--- a/freqtrade/exchange/exchange.py
+++ b/freqtrade/exchange/exchange.py
@@ -28,7 +28,7 @@
 from freqtrade.exchange.common import (API_FETCH_ORDER_RETRY_COUNT, BAD_EXCHANGES,
                                        EXCHANGE_HAS_OPTIONAL, EXCHANGE_HAS_REQUIRED, retrier,
                                        retrier_async)
-from freqtrade.misc import deep_merge_dicts, safe_value_fallback2
+from freqtrade.misc import chunks, deep_merge_dicts, safe_value_fallback2
 from freqtrade.plugins.pairlist.pairlist_helpers import expand_pairlist
 
 
@@ -1239,21 +1239,22 @@ async def _async_get_historic_ohlcv(self, pair: str, timeframe: str,
             pair, timeframe, since) for since in
             range(since_ms, arrow.utcnow().int_timestamp * 1000, one_call)]
 
-        results = await asyncio.gather(*input_coroutines, return_exceptions=True)
-
-        # Combine gathered results
         data: List = []
-        for res in results:
-            if isinstance(res, Exception):
-                logger.warning("Async code raised an exception: %s", res.__class__.__name__)
-                continue
-            # Deconstruct tuple if it's not an exception
-            p, _, new_data = res
-            if p == pair:
-                data.extend(new_data)
+        # Chunk requests into batches of 100 to avoid overwelming ccxt Throttling
+        for input_coro in chunks(input_coroutines, 100):
+
+            results = await asyncio.gather(*input_coro, return_exceptions=True)
+            for res in results:
+                if isinstance(res, Exception):
+                    logger.warning("Async code raised an exception: %s", res.__class__.__name__)
+                    continue
+                # Deconstruct tuple if it's not an exception
+                p, _, new_data = res
+                if p == pair:
+                    data.extend(new_data)
         # Sort data again after extending the result - above calls return in "async order"
         data = sorted(data, key=lambda x: x[0])
-        logger.info("Downloaded data for %s with length %s.", pair, len(data))
+        logger.info(f"Downloaded data for {pair} with length {len(data)}.")
         return data
 
     def refresh_latest_ohlcv(self, pair_list: ListPairsWithTimeframes, *,


Test patch:
diff --git a/tests/exchange/test_exchange.py b/tests/exchange/test_exchange.py
--- a/tests/exchange/test_exchange.py
+++ b/tests/exchange/test_exchange.py
@@ -1551,6 +1551,32 @@ async def mock_candle_hist(pair, timeframe, since_ms):
     assert 'high' in ret.columns
 
 
+@pytest.mark.asyncio
+@pytest.mark.parametrize("exchange_name", EXCHANGES)
+async def test__async_get_historic_ohlcv(default_conf, mocker, caplog, exchange_name):
+    ohlcv = [
+        [
+            int((datetime.now(timezone.utc).timestamp() - 1000) * 1000),
+            1,  # open
+            2,  # high
+            3,  # low
+            4,  # close
+            5,  # volume (in quote currency)
+        ]
+    ]
+    exchange = get_patched_exchange(mocker, default_conf, id=exchange_name)
+    # Monkey-patch async function
+    exchange._api_async.fetch_ohlcv = get_mock_coro(ohlcv)
+
+    pair = 'ETH/USDT'
+    res = await exchange._async_get_historic_ohlcv(pair, "5m",
+                                                   1500000000000, is_new_pair=False)
+    # Call with very old timestamp - causes tons of requests
+    assert exchange._api_async.fetch_ohlcv.call_count > 200
+    assert res[0] == ohlcv[0]
+    assert log_has_re(r'Downloaded data for .* with length .*\.', caplog)
+
+
 def test_refresh_latest_ohlcv(mocker, default_conf, caplog) -> None:
     ohlcv = [
         [