Repository: freqtrade/freqtrade

Issue:
Exception: volume is not double
## Step 1: Have you search for this issue before posting it?

Yes and talked about it on slack.

## Step 2: Describe your environment

  * Operating system: Ubuntu 19.10
  * Python Version: Python 3.7.5
  * CCXT version: ccxt==1.22.61
  * Branch: Develop
  * Last Commit ID: 43add0b1594f03379e61ef014b80a0a2723914de
  
## Step 3: Describe the problem:

When running hyperopt using downloaded data an exception will be thown.
**Exception: volume is not double**

When doing some troubleshooting it only affects some of the downloaded data (not all pairs)
XRP/BTC
```
date      datetime64[ns, UTC]
open                  float64
high                  float64
low                   float64
close                 float64
volume                  int64
dtype: object
```
### Steps to reproduce:

  1. Not sure :-/
  2. The main indicator that throws error is MFI (dataframe['mfi'] = ta.MFI(dataframe))
  
### Observed Results:

Unable to run hyperopt

### Relevant code exceptions or logs:

  ```
2020-02-20 22:47:28,025 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/opt/tradebot/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 20, in wrapper
    for arg in chain(args, kwargs.values())
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/freqtrade/freqtrade/commands/optimize_commands.py", line 76, in start_hyperopt
    hyperopt.start()
  File "/freqtrade/freqtrade/optimize/hyperopt.py", line 488, in start
    preprocessed = self.backtesting.strategy.tickerdata_to_dataframe(data)
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in tickerdata_to_dataframe
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in <dictcomp>
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 464, in advise_indicators
    return self.populate_indicators(dataframe, metadata)
  File "/freqtrade/user_data/strategies/MyDev.py", line 205, in populate_indicators
    dataframe['mfi'] = ta.MFI(dataframe)
  File "talib/_abstract.pxi", line 352, in talib._ta_lib.Function.__call__
  File "talib/_abstract.pxi", line 383, in talib._ta_lib.Function.__call_function
  File "/opt/tradebot/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 24, in wrapper
    return func(*args, **kwargs)
  File "talib/_func.pxi", line 8454, in talib._ta_lib.MFI
Exception: volume is not double

  ```

Exception: volume is not double
## Step 1: Have you search for this issue before posting it?

Yes and talked about it on slack.

## Step 2: Describe your environment

  * Operating system: Ubuntu 19.10
  * Python Version: Python 3.7.5
  * CCXT version: ccxt==1.22.61
  * Branch: Develop
  * Last Commit ID: 43add0b1594f03379e61ef014b80a0a2723914de
  
## Step 3: Describe the problem:

When running hyperopt using downloaded data an exception will be thown.
**Exception: volume is not double**

When doing some troubleshooting it only affects some of the downloaded data (not all pairs)
XRP/BTC
```
date      datetime64[ns, UTC]
open                  float64
high                  float64
low                   float64
close                 float64
volume                  int64
dtype: object
```
### Steps to reproduce:

  1. Not sure :-/
  2. The main indicator that throws error is MFI (dataframe['mfi'] = ta.MFI(dataframe))
  
### Observed Results:

Unable to run hyperopt

### Relevant code exceptions or logs:

  ```
2020-02-20 22:47:28,025 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/opt/tradebot/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 20, in wrapper
    for arg in chain(args, kwargs.values())
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/freqtrade/freqtrade/commands/optimize_commands.py", line 76, in start_hyperopt
    hyperopt.start()
  File "/freqtrade/freqtrade/optimize/hyperopt.py", line 488, in start
    preprocessed = self.backtesting.strategy.tickerdata_to_dataframe(data)
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in tickerdata_to_dataframe
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in <dictcomp>
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 464, in advise_indicators
    return self.populate_indicators(dataframe, metadata)
  File "/freqtrade/user_data/strategies/MyDev.py", line 205, in populate_indicators
    dataframe['mfi'] = ta.MFI(dataframe)
  File "talib/_abstract.pxi", line 352, in talib._ta_lib.Function.__call__
  File "talib/_abstract.pxi", line 383, in talib._ta_lib.Function.__call_function
  File "/opt/tradebot/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 24, in wrapper
    return func(*args, **kwargs)
  File "talib/_func.pxi", line 8454, in talib._ta_lib.MFI
Exception: volume is not double

  ```

Exception: volume is not double
## Step 1: Have you search for this issue before posting it?

Yes and talked about it on slack.

## Step 2: Describe your environment

  * Operating system: Ubuntu 19.10
  * Python Version: Python 3.7.5
  * CCXT version: ccxt==1.22.61
  * Branch: Develop
  * Last Commit ID: 43add0b1594f03379e61ef014b80a0a2723914de
  
## Step 3: Describe the problem:

When running hyperopt using downloaded data an exception will be thown.
**Exception: volume is not double**

When doing some troubleshooting it only affects some of the downloaded data (not all pairs)
XRP/BTC
```
date      datetime64[ns, UTC]
open                  float64
high                  float64
low                   float64
close                 float64
volume                  int64
dtype: object
```
### Steps to reproduce:

  1. Not sure :-/
  2. The main indicator that throws error is MFI (dataframe['mfi'] = ta.MFI(dataframe))
  
### Observed Results:

Unable to run hyperopt

### Relevant code exceptions or logs:

  ```
2020-02-20 22:47:28,025 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/opt/tradebot/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 20, in wrapper
    for arg in chain(args, kwargs.values())
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/freqtrade/freqtrade/commands/optimize_commands.py", line 76, in start_hyperopt
    hyperopt.start()
  File "/freqtrade/freqtrade/optimize/hyperopt.py", line 488, in start
    preprocessed = self.backtesting.strategy.tickerdata_to_dataframe(data)
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in tickerdata_to_dataframe
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in <dictcomp>
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 464, in advise_indicators
    return self.populate_indicators(dataframe, metadata)
  File "/freqtrade/user_data/strategies/MyDev.py", line 205, in populate_indicators
    dataframe['mfi'] = ta.MFI(dataframe)
  File "talib/_abstract.pxi", line 352, in talib._ta_lib.Function.__call__
  File "talib/_abstract.pxi", line 383, in talib._ta_lib.Function.__call_function
  File "/opt/tradebot/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 24, in wrapper
    return func(*args, **kwargs)
  File "talib/_func.pxi", line 8454, in talib._ta_lib.MFI
Exception: volume is not double

  ```



Discussion:
tried to delete downloaded data and re-download. Same issue
Thanks,

Similar was fixed an year ago: https://github.com/freqtrade/freqtrade/pull/1546

Either the fix was missing with later changes, or it was only applicable for non-stored dataframes, used during live/dry runs (so the same should be applied to downloaded data or during their load)

Still get same error message :-(

  * Last Commit ID: 430f53ca118190b1bf02a720cc9be3e3ba036dea

**What was done:**
- Updated to latest devel version
- Deleted all downloaded data
- re-downloaded data
- tested hyperopt

**Error:**
```
2020-02-22 16:15:33,158 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/opt/tradebot/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 20, in wrapper
    for arg in chain(args, kwargs.values())
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/freqtrade/freqtrade/commands/optimize_commands.py", line 76, in start_hyperopt
    hyperopt.start()
  File "/freqtrade/freqtrade/optimize/hyperopt.py", line 488, in start
    preprocessed = self.backtesting.strategy.tickerdata_to_dataframe(data)
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in tickerdata_to_dataframe
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in <dictcomp>
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 462, in advise_indicators
    return self.populate_indicators(dataframe)  # type: ignore
  File "/freqtrade/user_data/strategies/MyDev.py", line 218, in populate_indicators
    dataframe['mfi'] = ta.MFI(dataframe)
  File "talib/_abstract.pxi", line 352, in talib._ta_lib.Function.__call__
  File "talib/_abstract.pxi", line 383, in talib._ta_lib.Function.__call_function
  File "/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 24, in wrapper
    return func(*args, **kwargs)
  File "talib/_func.pxi", line 8454, in talib._ta_lib.MFI
Exception: volume is not double
```
(an organizational point)

@xmatthias strange, @Fredrik81 shown me a screenshot, he does not have a 'Reopen issue' button near green "Comment"... Are users able to reopen issues at all? There should be a configuration option for the repo for this...

(Added: I found the description for workflow implemented in github: https://stackoverflow.com/questions/21333654/how-to-re-open-an-issue-in-github/21333938#21333938

* you can re-open your own issues *if you closed them yourself
* you cannot re-open your own issues if a repo collaborator closed them

Seems it's not customizable...)

back to the issue itself:

I do not why, but this:
```
        pairdata = read_json(filename, orient='values', dtype=False)
```
resolves the issue (volumes are float64)...
Volumes are float in the json datatypes, but `dtype: dict` converts them to ints... Very strange behavior...

the problem is that columns after read_json do not have names 'date', 'close', etc, so `dtypes=` does not work...

tried to delete downloaded data and re-download. Same issue
Thanks,

Similar was fixed an year ago: https://github.com/freqtrade/freqtrade/pull/1546

Either the fix was missing with later changes, or it was only applicable for non-stored dataframes, used during live/dry runs (so the same should be applied to downloaded data or during their load)

Still get same error message :-(

  * Last Commit ID: 430f53ca118190b1bf02a720cc9be3e3ba036dea

**What was done:**
- Updated to latest devel version
- Deleted all downloaded data
- re-downloaded data
- tested hyperopt

**Error:**
```
2020-02-22 16:15:33,158 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/opt/tradebot/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 20, in wrapper
    for arg in chain(args, kwargs.values())
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/freqtrade/freqtrade/commands/optimize_commands.py", line 76, in start_hyperopt
    hyperopt.start()
  File "/freqtrade/freqtrade/optimize/hyperopt.py", line 488, in start
    preprocessed = self.backtesting.strategy.tickerdata_to_dataframe(data)
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in tickerdata_to_dataframe
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in <dictcomp>
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 462, in advise_indicators
    return self.populate_indicators(dataframe)  # type: ignore
  File "/freqtrade/user_data/strategies/MyDev.py", line 218, in populate_indicators
    dataframe['mfi'] = ta.MFI(dataframe)
  File "talib/_abstract.pxi", line 352, in talib._ta_lib.Function.__call__
  File "talib/_abstract.pxi", line 383, in talib._ta_lib.Function.__call_function
  File "/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 24, in wrapper
    return func(*args, **kwargs)
  File "talib/_func.pxi", line 8454, in talib._ta_lib.MFI
Exception: volume is not double
```
(an organizational point)

@xmatthias strange, @Fredrik81 shown me a screenshot, he does not have a 'Reopen issue' button near green "Comment"... Are users able to reopen issues at all? There should be a configuration option for the repo for this...

(Added: I found the description for workflow implemented in github: https://stackoverflow.com/questions/21333654/how-to-re-open-an-issue-in-github/21333938#21333938

* you can re-open your own issues *if you closed them yourself
* you cannot re-open your own issues if a repo collaborator closed them

Seems it's not customizable...)

back to the issue itself:

I do not why, but this:
```
        pairdata = read_json(filename, orient='values', dtype=False)
```
resolves the issue (volumes are float64)...
Volumes are float in the json datatypes, but `dtype: dict` converts them to ints... Very strange behavior...

the problem is that columns after read_json do not have names 'date', 'close', etc, so `dtypes=` does not work...

tried to delete downloaded data and re-download. Same issue
Thanks,

Similar was fixed an year ago: https://github.com/freqtrade/freqtrade/pull/1546

Either the fix was missing with later changes, or it was only applicable for non-stored dataframes, used during live/dry runs (so the same should be applied to downloaded data or during their load)

Still get same error message :-(

  * Last Commit ID: 430f53ca118190b1bf02a720cc9be3e3ba036dea

**What was done:**
- Updated to latest devel version
- Deleted all downloaded data
- re-downloaded data
- tested hyperopt

**Error:**
```
2020-02-22 16:15:33,158 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/opt/tradebot/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 20, in wrapper
    for arg in chain(args, kwargs.values())
StopIteration

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/freqtrade/freqtrade/main.py", line 36, in main
    return_code = args['func'](args)
  File "/freqtrade/freqtrade/commands/optimize_commands.py", line 76, in start_hyperopt
    hyperopt.start()
  File "/freqtrade/freqtrade/optimize/hyperopt.py", line 488, in start
    preprocessed = self.backtesting.strategy.tickerdata_to_dataframe(data)
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in tickerdata_to_dataframe
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 448, in <dictcomp>
    for pair, pair_data in tickerdata.items()}
  File "/freqtrade/freqtrade/strategy/interface.py", line 462, in advise_indicators
    return self.populate_indicators(dataframe)  # type: ignore
  File "/freqtrade/user_data/strategies/MyDev.py", line 218, in populate_indicators
    dataframe['mfi'] = ta.MFI(dataframe)
  File "talib/_abstract.pxi", line 352, in talib._ta_lib.Function.__call__
  File "talib/_abstract.pxi", line 383, in talib._ta_lib.Function.__call_function
  File "/freqtrade/.env/lib/python3.7/site-packages/talib/__init__.py", line 24, in wrapper
    return func(*args, **kwargs)
  File "talib/_func.pxi", line 8454, in talib._ta_lib.MFI
Exception: volume is not double
```
(an organizational point)

@xmatthias strange, @Fredrik81 shown me a screenshot, he does not have a 'Reopen issue' button near green "Comment"... Are users able to reopen issues at all? There should be a configuration option for the repo for this...

(Added: I found the description for workflow implemented in github: https://stackoverflow.com/questions/21333654/how-to-re-open-an-issue-in-github/21333938#21333938

* you can re-open your own issues *if you closed them yourself
* you cannot re-open your own issues if a repo collaborator closed them

Seems it's not customizable...)

back to the issue itself:

I do not why, but this:
```
        pairdata = read_json(filename, orient='values', dtype=False)
```
resolves the issue (volumes are float64)...
Volumes are float in the json datatypes, but `dtype: dict` converts them to ints... Very strange behavior...

the problem is that columns after read_json do not have names 'date', 'close', etc, so `dtypes=` does not work...


Resolving patch:
diff --git a/freqtrade/data/history/jsondatahandler.py b/freqtrade/data/history/jsondatahandler.py
--- a/freqtrade/data/history/jsondatahandler.py
+++ b/freqtrade/data/history/jsondatahandler.py
@@ -69,10 +69,10 @@ def _ohlcv_load(self, pair: str, timeframe: str,
         filename = self._pair_data_filename(self._datadir, pair, timeframe)
         if not filename.exists():
             return DataFrame(columns=self._columns)
-        pairdata = read_json(filename, orient='values',
-                             dtype={'open': 'float', 'high': 'float',
-                                    'low': 'float', 'close': 'float', 'volume': 'float'})
+        pairdata = read_json(filename, orient='values')
         pairdata.columns = self._columns
+        pairdata = pairdata.astype(dtype={'open': 'float', 'high': 'float',
+                                          'low': 'float', 'close': 'float', 'volume': 'float'})
         pairdata['date'] = to_datetime(pairdata['date'],
                                        unit='ms',
                                        utc=True,


Test patch: