Repository: freqtrade/freqtrade

Issue:
`processed` dict in hyperopt loss doesn't contain the dataframes
## Describe your environment

  * Operating system: MacOS 12.0.1 
  * Python Version: n/a
  * CCXT version:  n/a)
  * Freqtrade Version: freqtrade docker-bb29c444

## Describe the problem:

*Explain the problem you have encountered*
The dict `processed` passed to `hyperopt_loss_function()` doesn't contain any dataframes 

### Steps to reproduce:

  1.  create hyperopt custom loss in `user_data/hyperopts`
  2. add `print(processed)` within `hyperopt_loss_function`
  3. run
  ```
hyperopt
      --strategy MyStrategy
      --timerange 20210501-
      --config /freqtrade/user_data/configs/config.json
      --spaces buy sell
      --epochs 100
      --hyperopt-loss MyLoss
      --job-workers -1
      -p BTC/USDT 
      --print-all
```
  
### Observed Results:

  * This is the result I get when I run `print(processed)` inside `hyperopt_loss_function`:
  ```
2022-01-06 19:55:57,425 - freqtrade.optimize.hyperopt - INFO - Effective number of parallel workers used: 16
{'BTC/USDT': None}
{'BTC/USDT': None}
{'BTC/USDT': None}
{'BTC/USDT': None}
{'BTC/USDT': None}
{'BTC/USDT': None}
{'BTC/USDT': None}
```

  * Expected result:
 ```
{'BTC/USDT': pd.DataFrame}
```

- It works if I load the file manually like:
```
def hyperopt_loss_function(self, results: DataFrame, trade_count: int,
                               min_date: datetime, max_date: datetime,
                               config: Dict, processed: Dict[str, DataFrame],
                               backtest_stats: Dict[str, Any],
                               *args, **kwargs) -> float:
        print(processed)
        with open('user_data/hyperopt_results/hyperopt_tickerdata.pkl', 'rb') as f:
            processed = load(f, mmap_mode='r')
            print(processed)
```
![image](https://user-images.githubusercontent.com/45588475/148460359-31a2de0b-54c1-4f79-bb12-076c844b0237.png)

`processed` dict in hyperopt loss doesn't contain the dataframes
## Describe your environment

  * Operating system: MacOS 12.0.1 
  * Python Version: n/a
  * CCXT version:  n/a)
  * Freqtrade Version: freqtrade docker-bb29c444

## Describe the problem:

*Explain the problem you have encountered*
The dict `processed` passed to `hyperopt_loss_function()` doesn't contain any dataframes 

### Steps to reproduce:

  1.  create hyperopt custom loss in `user_data/hyperopts`
  2. add `print(processed)` within `hyperopt_loss_function`
  3. run
  ```
hyperopt
      --strategy MyStrategy
      --timerange 20210501-
      --config /freqtrade/user_data/configs/config.json
      --spaces buy sell
      --epochs 100
      --hyperopt-loss MyLoss
      --job-workers -1
      -p BTC/USDT 
      --print-all
```
  
### Observed Results:

  * This is the result I get when I run `print(processed)` inside `hyperopt_loss_function`:
  ```
2022-01-06 19:55:57,425 - freqtrade.optimize.hyperopt - INFO - Effective number of parallel workers used: 16
{'BTC/USDT': None}
{'BTC/USDT': None}
{'BTC/USDT': None}
{'BTC/USDT': None}
{'BTC/USDT': None}
{'BTC/USDT': None}
{'BTC/USDT': None}
```

  * Expected result:
 ```
{'BTC/USDT': pd.DataFrame}
```

- It works if I load the file manually like:
```
def hyperopt_loss_function(self, results: DataFrame, trade_count: int,
                               min_date: datetime, max_date: datetime,
                               config: Dict, processed: Dict[str, DataFrame],
                               backtest_stats: Dict[str, Any],
                               *args, **kwargs) -> float:
        print(processed)
        with open('user_data/hyperopt_results/hyperopt_tickerdata.pkl', 'rb') as f:
            processed = load(f, mmap_mode='r')
            print(processed)
```
![image](https://user-images.githubusercontent.com/45588475/148460359-31a2de0b-54c1-4f79-bb12-076c844b0237.png)



Discussion:
would you mind doing the same test with 2021.12? (just change the docker-tag to `:2021.12`).

i have a pretty strong suspicion what caused this (#6133) - and it seems like your setup is suited well to quickly test this

You should also treat your workaround with caution - as it will not include the buy signal (which processed actually should).
This CAN work correctly - but most likely will not (depending on what you're trying to do in your loss function).
Yep it works! It's definitely related
would you mind doing the same test with 2021.12? (just change the docker-tag to `:2021.12`).

i have a pretty strong suspicion what caused this (#6133) - and it seems like your setup is suited well to quickly test this

You should also treat your workaround with caution - as it will not include the buy signal (which processed actually should).
This CAN work correctly - but most likely will not (depending on what you're trying to do in your loss function).
Yep it works! It's definitely related

Resolving patch:
diff --git a/freqtrade/optimize/backtesting.py b/freqtrade/optimize/backtesting.py
--- a/freqtrade/optimize/backtesting.py
+++ b/freqtrade/optimize/backtesting.py
@@ -270,8 +270,8 @@ def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tupl
             df_analyzed = self.strategy.advise_sell(
                 self.strategy.advise_buy(pair_data, {'pair': pair}), {'pair': pair}).copy()
             # Trim startup period from analyzed dataframe
-            df_analyzed = trim_dataframe(df_analyzed, self.timerange,
-                                         startup_candles=self.required_startup)
+            df_analyzed = processed[pair] = pair_data = trim_dataframe(
+                df_analyzed, self.timerange, startup_candles=self.required_startup)
             # To avoid using data from future, we use buy/sell signals shifted
             # from the previous candle
             df_analyzed.loc[:, 'buy'] = df_analyzed.loc[:, 'buy'].shift(1)
@@ -287,9 +287,6 @@ def _get_ohlcv_as_lists(self, processed: Dict[str, DataFrame]) -> Dict[str, Tupl
             # Convert from Pandas to list for performance reasons
             # (Looping Pandas is slow.)
             data[pair] = df_analyzed[headers].values.tolist()
-
-            # Do not hold on to old data to reduce memory usage
-            processed[pair] = pair_data = None
         return data
 
     def _get_close_rate(self, sell_row: Tuple, trade: LocalTrade, sell: SellCheckTuple,


Test patch: