Repository: freqtrade/freqtrade

Issue:
Strange results for download-data
I recalled #2716, tried to reproduce and obtained very strange results:

* remove config.json
* then run:
```
$ freqtrade download-data --exchange gemini --pairs 'BTC/USD' 'LTC/USD' 'BCH/USD' 'ETH/USD' -t 1h --days 5 -v
2020-02-08 03:48:33,819 - freqtrade.loggers - INFO - Verbosity set to 1
2020-02-08 03:48:33,819 - freqtrade.configuration.configuration - INFO - Using exchange gemini
2020-02-08 03:48:33,819 - freqtrade.configuration.configuration - INFO - Using user-data directory: /home/user/freqtrade-wrk/github-hroff-1902/freqtrade/user_data ...
2020-02-08 03:48:33,819 - freqtrade.configuration.configuration - INFO - Using data directory: /home/user/freqtrade-wrk/github-hroff-1902/freqtrade/user_data/data/gemini ...
2020-02-08 03:48:33,819 - freqtrade.configuration.configuration - INFO - Using pairs ['BTC/USD', 'LTC/USD', 'BCH/USD', 'ETH/USD']
2020-02-08 03:48:33,820 - freqtrade.configuration.configuration - INFO - timeframes --timeframes: ['1h']
2020-02-08 03:48:33,820 - freqtrade.configuration.configuration - INFO - Detected --days: 5
2020-02-08 03:48:33,820 - freqtrade.configuration.check_exchange - INFO - Checking exchange...
2020-02-08 03:48:33,820 - freqtrade.configuration.check_exchange - WARNING - Exchange "gemini" is known to the the ccxt library, available for the bot, but not officially supported by the Freqtrade development team. It may work flawlessly (please report back) or have serious issues. Use it at your own discretion.
2020-02-08 03:48:33,820 - freqtrade.configuration.deprecated_settings - WARNING - DEPRECATED: Pairlists must be defined explicitly in the future.Defaulting to StaticPairList for now.
2020-02-08 03:48:33,820 - freqtrade.configuration.config_validation - INFO - Validating configuration ...
2020-02-08 03:48:33,824 - freqtrade.commands.data_commands - INFO - About to download pairs: ['BTC/USD', 'LTC/USD', 'BCH/USD', 'ETH/USD'], intervals: ['1h'] to /home/user/freqtrade-wrk/github-hroff-1902/freqtrade/user_data/data/gemini
2020-02-08 03:48:33,824 - freqtrade.resolvers.exchange_resolver - INFO - No Gemini specific subclass found. Using the generic class instead.
2020-02-08 03:48:33,824 - freqtrade.exchange.exchange - INFO - Instance is running with dry_run enabled
2020-02-08 03:48:33,826 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True}
2020-02-08 03:48:33,826 - asyncio - DEBUG - Using selector: EpollSelector
2020-02-08 03:48:33,828 - freqtrade.exchange.exchange - INFO - Using Exchange "Gemini"
2020-02-08 03:48:35,437 - chardet.charsetprober - DEBUG - utf-8  confidence = 0.99
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - SHIFT_JIS Japanese confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - EUC-JP Japanese confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - GB2312 Chinese confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - EUC-KR Korean confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - CP949 Korean confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - Big5 Chinese confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - EUC-TW Taiwan confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - windows-1251 Russian confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - KOI8-R Russian confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - ISO-8859-5 Russian confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - MacCyrillic Russian confidence = 0.0
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - IBM866 Russian confidence = 0.0
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - IBM855 Russian confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - ISO-8859-7 Greek confidence = 0.0
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - windows-1253 Greek confidence = 0.0
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - ISO-8859-5 Bulgairan confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - windows-1251 Bulgarian confidence = 0.01
2020-02-08 03:48:35,438 - chardet.charsetprober - DEBUG - TIS-620 Thai confidence = 0.0
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - ISO-8859-9 Turkish confidence = 0.6551158277756192
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - windows-1255 Hebrew confidence = 0.0
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - windows-1255 Hebrew confidence = 0.0
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - windows-1255 Hebrew confidence = 0.0
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - utf-8  confidence = 0.99
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - SHIFT_JIS Japanese confidence = 0.01
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - EUC-JP Japanese confidence = 0.01
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - GB2312 Chinese confidence = 0.01
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - EUC-KR Korean confidence = 0.01
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - CP949 Korean confidence = 0.01
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - Big5 Chinese confidence = 0.01
2020-02-08 03:48:35,439 - chardet.charsetprober - DEBUG - EUC-TW Taiwan confidence = 0.01
2020-02-08 03:48:35,441 - freqtrade - ERROR -  is not available as stake on Gemini. Available currencies are: BCH, BTC, ETH, LTC, USD
2020-02-08 03:48:35,441 - freqtrade.exchange.exchange - DEBUG - Exchange object destroyed, closing async loop
```

-- note 1) strange messages from "chardet.charsetprober" module;  2) empty "something" between "ERROR -  is not available". 3) strange module "freqtrade" for this message...

download-data validates whitelist when --pairs is used
The download-data subcommands validates whitelist when `--pairs` is used:

```
$ freqtrade download-data -c config.json --exchange kraken --pairs BTC/USD -t 1d --days 100 -v
...
2019-12-17 20:07:49,885 - freqtrade - ERROR - Pair NXT/BTC is not available on Kraken. Please remove NXT/BTC from your whitelist.
```

Expected behavior: ignore pair_whitelist, work only with pairs set in the cli.



Discussion:
> chardet.charsetprober

 ... this is a dependency of requests ... i suspect it's not lear which charset the return value is, therefore requests (called from ccxt) needs to "test" this.


I'll take a look at the "real" problem - which is that stake_currency is not set - which was (probably) introduced by removing that default from `MINIMAL_CONFIG`.
A solution to that is not as simple as it looks like ...

the main problem is that we've got 2 versions of the pairlist - `config['pairs']` and `config['exchange']['pair_whitelist']` - both of which can exist at the same time - with different contents.

in theory, we can fix this rather easily by copying `config['pairs']` into pair_whitelist ... but that would change the behaviour of a custom configuration file which specifies both lists (currently regular trading would use 'pair_whitelist' - but in the future it would use 'pairs' instead if that was specified).

 

Resolving patch:
diff --git a/freqtrade/commands/data_commands.py b/freqtrade/commands/data_commands.py
--- a/freqtrade/commands/data_commands.py
+++ b/freqtrade/commands/data_commands.py
@@ -37,7 +37,12 @@ def start_download_data(args: Dict[str, Any]) -> None:
     pairs_not_available: List[str] = []
 
     # Init exchange
-    exchange = ExchangeResolver.load_exchange(config['exchange']['name'], config)
+    exchange = ExchangeResolver.load_exchange(config['exchange']['name'], config, validate=False)
+    # Manual validations of relevant settings
+    exchange.validate_pairs(config['pairs'])
+    for timeframe in config['timeframes']:
+        exchange.validate_timeframes(timeframe)
+
     try:
 
         if config.get('download_trades'):


Test patch: