Repository: freqtrade/freqtrade

Issue:
download-data fails with difficult to interpret messages in certain cases
1)
```
(.env) user@nuc:~/freqtrade-wrk/github-hroff-1902/freqtrade$ freqtrade download-data
2019-09-06 01:00:01,335 - freqtrade.loggers - INFO - Verbosity set to 0
2019-09-06 01:00:01,336 - freqtrade.configuration.configuration - INFO - Dry run is enabled
2019-09-06 01:00:01,336 - freqtrade.configuration.configuration - INFO - Using DB: "sqlite://"
2019-09-06 01:00:01,336 - freqtrade.configuration.configuration - INFO - Using max_open_trades: None ...
2019-09-06 01:00:01,336 - freqtrade.configuration.configuration - INFO - Using user-data directory: /home/user/freqtrade-wrk/github-hroff-1902/freqtrade/user_data ...
2019-09-06 01:00:01,337 - freqtrade.configuration.configuration - INFO - Using data directory: /home/user/freqtrade-wrk/github-hroff-1902/freqtrade/user_data/data ...
2019-09-06 01:00:01,337 - freqtrade.configuration.configuration - INFO - timeframes --timeframes: ['1m', '5m']
2019-09-06 01:00:01,337 - freqtrade.configuration.check_exchange - INFO - Checking exchange...
2019-09-06 01:00:01,337 - freqtrade - ERROR - Exchange "" is not supported by ccxt and therefore not available for the bot.
The following exchanges are supported by ccxt: _1btcxe, acx, allcoin, anxpro, bcex, bequant, bibox, bigone, binance, binanceje, bit2c, bitbank, bitbay, bitfinex, bitfinex2, bitflyer, bitforex, bithumb, bitkk, bitlish, bitmart, bitmex, bitso, bitstamp, bitstamp1, bittrex, bitz, bl3p, bleutrade, braziliex, btcalpha, btcbox, btcchina, btcmarkets, btctradeim, btctradeua, btcturk, buda, bxinth, cex, chilebit, cobinhood, coinbase, coinbaseprime, coinbasepro, coincheck, coinegg, coinex, coinexchange, coinfalcon, coinfloor, coingi, coinmarketcap, coinmate, coinone, coinspot, cointiger, coolcoin, coss, crex24, crypton, deribit, digifinex, dsx, dx, ethfinex, exmo, exx, fcoin, fcoinjp, flowbtc, foxbit, fybse, gateio, gdax, gemini, hitbtc, hitbtc2, huobipro, huobiru, ice3x, independentreserve, indodax, itbit, kkex, kraken, kucoin, kucoin2, kuna, lakebtc, latoken, lbank, liquid, livecoin, luno, lykke, mandala, mercado, mixcoins, negociecoins, nova, oceanex, okcoincny, okcoinusd, okex, okex3, paymium, poloniex, rightbtc, southxchange, stronghold, surbitcoin, theocean, therock, tidebit, tidex, upbit, vaultoro, vbtc, virwox, xbtce, yobit, zaif, zb
```
-- Maybe it's covered by #2217  . If not, the message should not include `Exchange ""` but should be something like `Exchange is not specified.`

2) 
```
(.env) user@nuc:~/freqtrade-wrk/github-hroff-1902/freqtrade$ freqtrade download-data --exchange bittrex
2019-09-06 01:00:15,176 - freqtrade.loggers - INFO - Verbosity set to 0
2019-09-06 01:00:15,177 - freqtrade.configuration.configuration - INFO - Dry run is enabled
2019-09-06 01:00:15,177 - freqtrade.configuration.configuration - INFO - Using DB: "sqlite://"
2019-09-06 01:00:15,177 - freqtrade.configuration.configuration - INFO - Using max_open_trades: None ...
2019-09-06 01:00:15,177 - freqtrade.configuration.configuration - INFO - Using exchange bittrex
2019-09-06 01:00:15,178 - freqtrade.configuration.configuration - INFO - Using user-data directory: /home/user/freqtrade-wrk/github-hroff-1902/freqtrade/user_data ...
2019-09-06 01:00:15,178 - freqtrade.configuration.configuration - INFO - Using data directory: /home/user/freqtrade-wrk/github-hroff-1902/freqtrade/user_data/data/bittrex ...
2019-09-06 01:00:15,178 - freqtrade.configuration.configuration - INFO - timeframes --timeframes: ['1m', '5m']
2019-09-06 01:00:15,178 - freqtrade.configuration.check_exchange - INFO - Checking exchange...
2019-09-06 01:00:15,178 - freqtrade.configuration.check_exchange - INFO - Exchange "bittrex" is officially supported by the Freqtrade development team.
2019-09-06 01:00:15,179 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/user/freqtrade-wrk/github-hroff-1902/freqtrade/freqtrade/main.py", line 40, in main
    args.func(args)
  File "/home/user/freqtrade-wrk/github-hroff-1902/freqtrade/freqtrade/utils.py", line 75, in start_download_data
    logger.info(f'About to download pairs: {config["pairs"]}, '
KeyError: 'pairs'
```



Discussion:
Both fixed by #2217. (I wonder why 2) is also fixed. Pairs are set in the minimal config?)
Closing... 

Note to myself: Need to review it along with #2217 


I've renamed the issue now, since it's not broken, but in certain cases (no `--config`, no `--exchange xxx --pairs yyy`) the messages are not 100% clear.

the first message is a side-effect of checking exchanges against ccxt. 
if neihter `--exchange`, nor `--config` is available, then the bot does not know which exchange to use - and "" is in fact, not a valid exchange. While irritating, the message is technically correct.
Adding a default exchange to minimal config would fix this - but whichever you choose, it'll be incorrect for some users (in the future more, if we support more exchanges).

The fix #2217 adds is to load `config.json` if it exists (apparently it does for you / the affected user). 
Remove that and the problem will reappear. 

This leads to the 2nd problem, without having exchange, `pairs.json` won't be found (it looks in the wrong folder). Assuming that's not present, there is no information what pairs the user would like - that's required to correctly download data as well, so somehow needs to be passed in (as `--pairs`, `pairs.json` or `whitelist` in configuration).

I think I closed it after the merge of #2217 into develop. It was a late night...
@xmatthias people run very simple intuitive commands: `freqtrade download-data --exchange exchange` and `freqtrade download-data` and it does not work. This means that download-data is (was) broken.
I don't disagree with this fact, but using titles that explain the problem are easier to re-find  than "it's broken" ...

Additionally, i would not consider this closed yet to be honest, since it'll produce the same errors if `config.json` does not exist in the current directory.



Resolving patch:
diff --git a/freqtrade/configuration/check_exchange.py b/freqtrade/configuration/check_exchange.py
--- a/freqtrade/configuration/check_exchange.py
+++ b/freqtrade/configuration/check_exchange.py
@@ -27,6 +27,14 @@ def check_exchange(config: Dict[str, Any], check_for_bad: bool = True) -> bool:
     logger.info("Checking exchange...")
 
     exchange = config.get('exchange', {}).get('name').lower()
+    if not exchange:
+        raise OperationalException(
+            f'This command requires a configured exchange. You should either use '
+            f'`--exchange <exchange_name>` or specify a configuration file via `--config`.\n'
+            f'The following exchanges are supported by ccxt: '
+            f'{", ".join(available_exchanges())}'
+        )
+
     if not is_exchange_available(exchange):
         raise OperationalException(
                 f'Exchange "{exchange}" is not supported by ccxt '
diff --git a/freqtrade/utils.py b/freqtrade/utils.py
--- a/freqtrade/utils.py
+++ b/freqtrade/utils.py
@@ -5,6 +5,7 @@
 
 import arrow
 
+from freqtrade import OperationalException
 from freqtrade.configuration import Configuration, TimeRange
 from freqtrade.configuration.directory_operations import create_userdata_dir
 from freqtrade.data.history import refresh_backtest_ohlcv_data
@@ -70,6 +71,11 @@ def start_download_data(args: Dict[str, Any]) -> None:
         time_since = arrow.utcnow().shift(days=-config['days']).strftime("%Y%m%d")
         timerange = TimeRange.parse_timerange(f'{time_since}-')
 
+    if 'pairs' not in config:
+        raise OperationalException(
+            "Downloading data requires a list of pairs. "
+            "Please check the documentation on how to configure this.")
+
     dl_path = Path(config['datadir'])
     logger.info(f'About to download pairs: {config["pairs"]}, '
                 f'intervals: {config["timeframes"]} to {dl_path}')


Test patch:
diff --git a/tests/test_configuration.py b/tests/test_configuration.py
--- a/tests/test_configuration.py
+++ b/tests/test_configuration.py
@@ -546,6 +546,13 @@ def test_check_exchange(default_conf, caplog) -> None:
     default_conf['runmode'] = RunMode.PLOT
     assert check_exchange(default_conf)
 
+    # Test no exchange...
+    default_conf.get('exchange').update({'name': ''})
+    default_conf['runmode'] = RunMode.OTHER
+    with pytest.raises(OperationalException,
+                       match=r'This command requires a configured exchange.*'):
+        check_exchange(default_conf)
+
 
 def test_cli_verbose_with_params(default_conf, mocker, caplog) -> None:
     patched_configuration_load_config_file(mocker, default_conf)
diff --git a/tests/test_utils.py b/tests/test_utils.py
--- a/tests/test_utils.py
+++ b/tests/test_utils.py
@@ -1,12 +1,14 @@
 import re
+from pathlib import Path
 from unittest.mock import MagicMock, PropertyMock
 
 import pytest
 
+from freqtrade import OperationalException
 from freqtrade.state import RunMode
-from tests.conftest import get_args, log_has, patch_exchange
 from freqtrade.utils import (setup_utils_configuration, start_create_userdir,
                              start_download_data, start_list_exchanges)
+from tests.conftest import get_args, log_has, patch_exchange
 
 
 def test_setup_utils_configuration():
@@ -103,3 +105,37 @@ def test_download_data_no_markets(mocker, caplog):
     start_download_data(get_args(args))
     assert dl_mock.call_args[1]['timerange'].starttype == "date"
     assert log_has("Pairs [ETH/BTC,XRP/BTC] not available on exchange binance.", caplog)
+
+
+def test_download_data_no_exchange(mocker, caplog):
+    mocker.patch('freqtrade.utils.refresh_backtest_ohlcv_data',
+                 MagicMock(return_value=["ETH/BTC", "XRP/BTC"]))
+    patch_exchange(mocker)
+    mocker.patch(
+        'freqtrade.exchange.Exchange.markets', PropertyMock(return_value={})
+    )
+    args = [
+        "download-data",
+        ]
+    with pytest.raises(OperationalException,
+                       match=r"This command requires a configured exchange.*"):
+        start_download_data(get_args(args))
+
+
+def test_download_data_no_pairs(mocker, caplog):
+    mocker.patch.object(Path, "exists", MagicMock(return_value=False))
+
+    mocker.patch('freqtrade.utils.refresh_backtest_ohlcv_data',
+                 MagicMock(return_value=["ETH/BTC", "XRP/BTC"]))
+    patch_exchange(mocker)
+    mocker.patch(
+        'freqtrade.exchange.Exchange.markets', PropertyMock(return_value={})
+    )
+    args = [
+        "download-data",
+        "--exchange",
+        "binance",
+    ]
+    with pytest.raises(OperationalException,
+                       match=r"Downloading data requires a list of pairs\..*"):
+        start_download_data(get_args(args))