Repository: freqtrade/freqtrade

Issue:
Docker-compose: Hyperopt - Fatal Exception - TypeError
## Step 1: Have you search for this issue before posting it?

If you have discovered a bug in the bot, please [search our issue tracker](https://github.com/freqtrade/freqtrade/issues?q=is%3Aissue). 
If it hasn't been reported, please create a new issue.

## Step 2: Describe your environment

  * Operating system: Linux
  * Python Version: 3.8.2 (conda-env)
  * CCXT version: 1.23.69
  * Branch: Master

## Step 3: Describe the problem:

The hyperopt module crashed because of a TypeError.

### Steps to reproduce:

  1. docker-compose run --rm freqtrade hyperopt --hyperopt CustomHyperopt --hyperopt-loss CustomHyperOptLoss --spaces all --timerange 20191011-20200119 --logfile /freqtrade/user_data/freqtrade_opt.log -e 100 --strategy CustomStrategy
  
### Observed Results:

  * What happened?
Hyperopt crashed

  * What did you expect to happen?
Run the hyperopt module with the provided parameters

### Relevant code exceptions or logs:

  ```
2020-03-09 22:42:15,822 - freqtrade.loggers - INFO - Verbosity set to 0
2020-03-09 22:42:15,822 - freqtrade.configuration.configuration - INFO - Using max_open_trades: 13 ...
2020-03-09 22:42:15,822 - freqtrade.configuration.configuration - INFO - Parameter --timerange detected: 20191011-20200119 ...
2020-03-09 22:42:15,822 - freqtrade.configuration.configuration - INFO - Using user-data directory: /freqtrade/user_data ...
2020-03-09 22:42:15,823 - freqtrade.configuration.configuration - INFO - Using data directory: /freqtrade/user_data/data/binance ...
2020-03-09 22:42:15,823 - freqtrade.configuration.configuration - INFO - Using Hyperopt class name: CustomHyperopt
2020-03-09 22:42:15,823 - freqtrade.configuration.configuration - INFO - Parameter --epochs detected ... Will run Hyperopt with for 100 epochs ...
2020-03-09 22:42:15,823 - freqtrade.configuration.configuration - INFO - Parameter -s/--spaces detected: ['all']
2020-03-09 22:42:15,823 - freqtrade.configuration.configuration - INFO - Parameter -j/--job-workers detected: -1
2020-03-09 22:42:15,823 - freqtrade.configuration.configuration - INFO - Parameter --min-trades detected: 1
2020-03-09 22:42:15,823 - freqtrade.configuration.configuration - INFO - Using Hyperopt loss class name: CustomHyperOptLoss
2020-03-09 22:42:15,824 - freqtrade.configuration.check_exchange - INFO - Checking exchange...
2020-03-09 22:42:15,824 - freqtrade.configuration.check_exchange - INFO - Exchange "binance" is officially supported by the Freqtrade development team.
2020-03-09 22:42:15,824 - freqtrade.configuration.configuration - INFO - Using pairlist from configuration.
2020-03-09 22:42:15,824 - freqtrade.configuration.config_validation - INFO - Validating configuration ...
2020-03-09 22:42:15,827 - freqtrade.commands.optimize_commands - INFO - Starting freqtrade in Hyperopt mode
2020-03-09 22:42:15,827 - filelock - INFO - Lock 140426008287024 acquired on /freqtrade/user_data/hyperopt.lock
2020-03-09 22:42:15,828 - freqtrade.exchange.exchange - INFO - Instance is running with dry_run enabled
2020-03-09 22:42:15,828 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True}
2020-03-09 22:42:15,832 - freqtrade.exchange.exchange - INFO - Applying additional ccxt config: {'enableRateLimit': True, 'rateLimit': 200}
2020-03-09 22:42:15,838 - freqtrade.exchange.exchange - INFO - Using Exchange "Binance"
2020-03-09 22:42:16,738 - freqtrade.resolvers.exchange_resolver - INFO - Using resolved exchange 'Binance'...
2020-03-09 22:42:16,770 - freqtrade.resolvers.iresolver - INFO - Using resolved strategy CustomStrategy from '/freqtrade/user_data/strategies/custom_strategy.py'...
2020-03-09 22:42:16,770 - freqtrade.resolvers.strategy_resolver - INFO - Override strategy 'ticker_interval' with value in config file: 15m.
2020-03-09 22:42:16,770 - freqtrade.resolvers.strategy_resolver - INFO - Override strategy 'trailing_stop' with value in config file: True.
2020-03-09 22:42:16,770 - freqtrade.resolvers.strategy_resolver - INFO - Override strategy 'stake_currency' with value in config file: BTC.
2020-03-09 22:42:16,771 - freqtrade.resolvers.strategy_resolver - INFO - Override strategy 'stake_amount' with value in config file: unlimited.
2020-03-09 22:42:16,771 - freqtrade.resolvers.strategy_resolver - INFO - Override strategy 'unfilledtimeout' with value in config file: {'buy': 10, 'sell': 30}.
2020-03-09 22:42:16,771 - freqtrade.resolvers.strategy_resolver - INFO - Override strategy 'use_sell_signal' with value in config file: True.
2020-03-09 22:42:16,771 - freqtrade.resolvers.strategy_resolver - INFO - Override strategy 'sell_profit_only' with value in config file: False.
2020-03-09 22:42:16,771 - freqtrade.resolvers.strategy_resolver - INFO - Override strategy 'ignore_roi_if_buy_signal' with value in config file: False.
2020-03-09 22:42:16,771 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using minimal_roi: {'0': 0.09638, '19': 0.03643, '69': 0.01923, '120': 0}
2020-03-09 22:42:16,771 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using ticker_interval: 15m
2020-03-09 22:42:16,771 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using stoploss: -0.36828
2020-03-09 22:42:16,771 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using trailing_stop: True
2020-03-09 22:42:16,771 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using trailing_stop_positive_offset: 0.0
2020-03-09 22:42:16,772 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using trailing_only_offset_is_reached: False
2020-03-09 22:42:16,772 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using process_only_new_candles: False
2020-03-09 22:42:16,772 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using order_types: {'buy': 'limit', 'sell': 'limit', 'stoploss': 'limit', 'stoploss_on_exchange': False}
2020-03-09 22:42:16,772 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using order_time_in_force: {'buy': 'gtc', 'sell': 'gtc'}
2020-03-09 22:42:16,772 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using stake_currency: BTC
2020-03-09 22:42:16,772 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using stake_amount: unlimited
2020-03-09 22:42:16,772 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using startup_candle_count: 0
2020-03-09 22:42:16,772 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using unfilledtimeout: {'buy': 10, 'sell': 30}
2020-03-09 22:42:16,772 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using use_sell_signal: True
2020-03-09 22:42:16,773 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using sell_profit_only: False
2020-03-09 22:42:16,773 - freqtrade.resolvers.strategy_resolver - INFO - Strategy using ignore_roi_if_buy_signal: False
2020-03-09 22:42:16,773 - freqtrade.configuration.config_validation - INFO - Validating configuration ...
2020-03-09 22:42:16,777 - freqtrade.resolvers.iresolver - INFO - Using resolved hyperopt CustomHyperopt from '/freqtrade/user_data/hyperopts/custom_hyperopt.py'...
2020-03-09 22:42:16,779 - freqtrade.resolvers.iresolver - INFO - Using resolved hyperoptloss CustomHyperOptLoss from '/freqtrade/user_data/hyperopts/custom_loss.py'...
2020-03-09 22:42:16,779 - freqtrade.optimize.hyperopt - INFO - Removing `/freqtrade/user_data/hyperopt_results/hyperopt_tickerdata.pkl`.
2020-03-09 22:42:16,780 - freqtrade.optimize.hyperopt - INFO - Using optimizer random state: 60654
2020-03-09 22:42:16,898 - freqtrade.data.converter - INFO - Missing data fillup for ETH/BTC: before: 9584 - after: 9601
2020-03-09 22:42:16,924 - freqtrade.data.history.idatahandler - WARNING - Missing data at start for pair BCH/BTC, data starts at 2019-11-28 10:00:00
2020-03-09 22:42:17,001 - freqtrade.data.converter - INFO - Missing data fillup for LTC/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,055 - freqtrade.data.converter - INFO - Missing data fillup for EOS/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,115 - freqtrade.data.converter - INFO - Missing data fillup for XRP/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,169 - freqtrade.data.converter - INFO - Missing data fillup for ETC/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,268 - freqtrade.data.converter - INFO - Missing data fillup for NEO/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,323 - freqtrade.data.converter - INFO - Missing data fillup for LINK/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,383 - freqtrade.data.converter - INFO - Missing data fillup for DASH/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,441 - freqtrade.data.converter - INFO - Missing data fillup for QTUM/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,504 - freqtrade.data.converter - INFO - Missing data fillup for ZEC/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,566 - freqtrade.data.converter - INFO - Missing data fillup for XTZ/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,622 - freqtrade.data.converter - INFO - Missing data fillup for ATOM/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,731 - freqtrade.data.converter - INFO - Missing data fillup for OMG/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,786 - freqtrade.data.converter - INFO - Missing data fillup for ONT/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,844 - freqtrade.data.converter - INFO - Missing data fillup for XMR/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,861 - freqtrade.data.history.idatahandler - WARNING - Missing data at start for pair OGN/BTC, data starts at 2020-01-09 03:00:00
2020-03-09 22:42:17,942 - freqtrade.data.converter - INFO - Missing data fillup for ALGO/BTC: before: 9584 - after: 9601
2020-03-09 22:42:17,995 - freqtrade.data.converter - INFO - Missing data fillup for DCR/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,052 - freqtrade.data.converter - INFO - Missing data fillup for BAT/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,149 - freqtrade.data.converter - INFO - Missing data fillup for WAVES/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,202 - freqtrade.data.converter - INFO - Missing data fillup for KNC/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,258 - freqtrade.data.converter - INFO - Missing data fillup for REP/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,311 - freqtrade.data.converter - INFO - Missing data fillup for BEAM/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,365 - freqtrade.data.converter - INFO - Missing data fillup for ZRX/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,420 - freqtrade.data.converter - INFO - Missing data fillup for HC/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,473 - freqtrade.data.converter - INFO - Missing data fillup for BTG/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,578 - freqtrade.data.converter - INFO - Missing data fillup for MCO/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,630 - freqtrade.data.converter - INFO - Missing data fillup for XZC/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,687 - freqtrade.data.converter - INFO - Missing data fillup for LSK/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,746 - freqtrade.data.converter - INFO - Missing data fillup for GAS/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,766 - freqtrade.data.history.idatahandler - WARNING - Missing data at start for pair FTT/BTC, data starts at 2019-12-20 03:00:00
2020-03-09 22:42:18,844 - freqtrade.data.converter - INFO - Missing data fillup for ZEN/BTC: before: 9584 - after: 9601
2020-03-09 22:42:18,857 - freqtrade.optimize.backtesting - INFO - Loading data from 2019-10-11T00:00:00+00:00 up to 2020-01-19T00:00:00+00:00 (100 days)..
2020-03-09 22:42:19,661 - freqtrade.optimize.hyperopt - INFO - Hyperopting with data from 2019-10-11T00:00:00+00:00 up to 2020-01-19T00:00:00+00:00 (100 days)..
2020-03-09 22:42:19,757 - freqtrade.optimize.hyperopt - INFO - Found 8 CPU cores. Let's make them scream!
2020-03-09 22:42:19,757 - freqtrade.optimize.hyperopt - INFO - Number of parallel jobs set as: -1
2020-03-09 22:42:19,781 - freqtrade.optimize.hyperopt - INFO - Effective number of parallel workers used: 8
2020-03-09 22:42:26,363 - freqtrade - ERROR - Fatal exception!
joblib.externals.loky.process_executor._RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 418, in _process_worker
    r = call_item()
  File "/usr/local/lib/python3.8/site-packages/joblib/externals/loky/process_executor.py", line 272, in __call__
    return self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 608, in __call__
    return self.func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/joblib/parallel.py", line 255, in __call__
    return [func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/joblib/parallel.py", line 255, in <listcomp>
    return [func(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/joblib/externals/loky/cloudpickle_wrapper.py", line 38, in __call__
    return self._obj(*args, **kwargs)
  File "/freqtrade/freqtrade/optimize/hyperopt.py", line 376, in generate_optimizer
    backtesting_results = self.backtesting.backtest(
  File "/freqtrade/freqtrade/optimize/backtesting.py", line 355, in backtest
    trade_entry = self._get_sell_trade_entry(pair, row, ticker[pair][indexes[pair]-1:],
  File "/freqtrade/freqtrade/optimize/backtesting.py", line 231, in _get_sell_trade_entry
    amount=stake_amount / buy_row.open,
TypeError: unsupported operand type(s) for /: 'str' and 'float'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/freqtrade/freqtrade/commands/optimize_commands.py", line 76, in start_hyperopt
    hyperopt.start()
  File "/freqtrade/freqtrade/optimize/hyperopt.py", line 524, in start
    f_val = self.run_optimizer_parallel(parallel, asked, i)
  File "/freqtrade/freqtrade/optimize/hyperopt.py", line 461, in run_optimizer_parallel
    return parallel(delayed(
  File "/usr/local/lib/python3.8/site-packages/joblib/parallel.py", line 1017, in __call__
    self.retrieve()
  File "/usr/local/lib/python3.8/site-packages/joblib/parallel.py", line 909, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/usr/local/lib/python3.8/site-packages/joblib/_parallel_backends.py", line 562, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 439, in result
    return self.__get_result()
  File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
    raise self._exception
TypeError: unsupported operand type(s) for /: 'str' and 'float'
  ```



Discussion:
`stake_amount="unlimited"` is not supported for neither backtesting nor hyperopt.

Please configure a value (e.g. stake_amount=100) - which fits both your stake-currency and your account - so values will be realistic.


hmm, why this is not validated in this case, as for Edge, where it prints that it rather needs unlimited value to be set? (should validation be added this way?)

(I thought in case of unlimited, the proper absolute stake amount was calculated (via dry-run wallet and max_open_trades) and used for backtesting...)
@xmatthias But this way you won't be able to hyperopt or backtest with dynamic stakes. Won't it be possible to use the dry wallet values instead? 
not at the moment, no - and unfortunately it's not just a simple division.

It would be wrong to simply take `dry_run_wallet` (say 100) and divide that by `max_open_trades` (say 3).
This would lead to a stake-amount of 33.3.

Now this is correct for the first 3 trades - however, after the first trade closed (either with profit, or loss) - the stake-amount would have to be recalculated to fit to these new parameters (so the 4th trade could open with 35 as the 1st trade was a win - or with 30 if the 1st trade was a loss).

It's called compounding - and that's what happens when using stake_amount=unlimited in both dry-run and live, so to properly simulate this - compounding would need to be implemented into backtesting - which will happen eventually - but until it is, stake_amount="unlimited" will remain unsupported.

> hmm, why this is not validated in this case, as for Edge, where it prints that it rather needs unlimited value to be set? (should validation be added this way?)

it is for backtesting - but apparently not for Hyperopt... (`optimize_commands.py` Line 20 ff)

Resolving patch:
diff --git a/freqtrade/commands/optimize_commands.py b/freqtrade/commands/optimize_commands.py
--- a/freqtrade/commands/optimize_commands.py
+++ b/freqtrade/commands/optimize_commands.py
@@ -17,10 +17,15 @@ def setup_optimize_configuration(args: Dict[str, Any], method: RunMode) -> Dict[
     """
     config = setup_utils_configuration(args, method)
 
-    if method == RunMode.BACKTEST:
-        if config['stake_amount'] == constants.UNLIMITED_STAKE_AMOUNT:
-            raise DependencyException('stake amount could not be "%s" for backtesting' %
-                                      constants.UNLIMITED_STAKE_AMOUNT)
+    no_unlimited_runmodes = {
+        RunMode.BACKTEST: 'backtesting',
+        RunMode.HYPEROPT: 'hyperoptimization',
+    }
+    if (method in no_unlimited_runmodes.keys() and
+            config['stake_amount'] == constants.UNLIMITED_STAKE_AMOUNT):
+        raise DependencyException(
+            f'The value of `stake_amount` cannot be set as "{constants.UNLIMITED_STAKE_AMOUNT}" '
+            f'for {no_unlimited_runmodes[method]}')
 
     return config
 


Test patch:
diff --git a/tests/optimize/test_backtesting.py b/tests/optimize/test_backtesting.py
--- a/tests/optimize/test_backtesting.py
+++ b/tests/optimize/test_backtesting.py
@@ -241,7 +241,7 @@ def test_setup_optimize_configuration_unlimited_stake_amount(mocker, default_con
         '--strategy', 'DefaultStrategy',
     ]
 
-    with pytest.raises(DependencyException, match=r'.*stake amount.*'):
+    with pytest.raises(DependencyException, match=r'.`stake_amount`.*'):
         setup_optimize_configuration(get_args(args), RunMode.BACKTEST)
 
 
diff --git a/tests/optimize/test_hyperopt.py b/tests/optimize/test_hyperopt.py
--- a/tests/optimize/test_hyperopt.py
+++ b/tests/optimize/test_hyperopt.py
@@ -10,10 +10,11 @@
 from arrow import Arrow
 from filelock import Timeout
 
+from freqtrade import constants
 from freqtrade.commands.optimize_commands import (setup_optimize_configuration,
                                                   start_hyperopt)
 from freqtrade.data.history import load_data
-from freqtrade.exceptions import OperationalException
+from freqtrade.exceptions import DependencyException, OperationalException
 from freqtrade.optimize.default_hyperopt import DefaultHyperOpt
 from freqtrade.optimize.default_hyperopt_loss import DefaultHyperOptLoss
 from freqtrade.optimize.hyperopt import Hyperopt
@@ -158,6 +159,21 @@ def test_setup_hyperopt_configuration_with_arguments(mocker, default_conf, caplo
     assert log_has('Parameter --print-all detected ...', caplog)
 
 
+def test_setup_hyperopt_configuration_unlimited_stake_amount(mocker, default_conf, caplog) -> None:
+    default_conf['stake_amount'] = constants.UNLIMITED_STAKE_AMOUNT
+
+    patched_configuration_load_config_file(mocker, default_conf)
+
+    args = [
+        'hyperopt',
+        '--config', 'config.json',
+        '--hyperopt', 'DefaultHyperOpt',
+    ]
+
+    with pytest.raises(DependencyException, match=r'.`stake_amount`.*'):
+        setup_optimize_configuration(get_args(args), RunMode.HYPEROPT)
+
+
 def test_hyperoptresolver(mocker, default_conf, caplog) -> None:
     patched_configuration_load_config_file(mocker, default_conf)