Repository: freqtrade/freqtrade

Issue:
Hyperopt memory leak catch
  1134/10000:      3 trades. Avg profit -0.77%. Total profit -0.02327087 ETH (  -2.32Σ%). Avg duration 1261.3 mins. Objective: 2.26564
........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................../home/dri/miner/freqtrade/.env/lib/python3.7/site-packages/joblib/externals/loky/process_executor.py:706: UserWarning: A worker stopped while some jobs were given to the executor. This can be caused by a too short worker timeout or by a memory leak.
  "timeout or by a memory leak.", UserWarning
2019-09-17 22:19:13,769 - freqtrade - ERROR - Fatal exception!
Traceback (most recent call last):
  File "/home/dri/miner/freqtrade/freqtrade/main.py", line 39, in main
    args['func'](args)
  File "/home/dri/miner/freqtrade/freqtrade/optimize/__init__.py", line 74, in start_hyperopt
    hyperopt.start()
  File "/home/dri/miner/freqtrade/freqtrade/optimize/hyperopt.py", line 404, in start
    f_val = self.run_optimizer_parallel(parallel, asked)
  File "/home/dri/miner/freqtrade/freqtrade/optimize/hyperopt.py", line 341, in run_optimizer_parallel
    wrap_non_picklable_objects(self.generate_optimizer))(v) for v in asked)
  File "/home/dri/miner/freqtrade/.env/lib/python3.7/site-packages/joblib/parallel.py", line 934, in __call__
    self.retrieve()
  File "/home/dri/miner/freqtrade/.env/lib/python3.7/site-packages/joblib/parallel.py", line 833, in retrieve
    self._output.extend(job.get(timeout=self.timeout))
  File "/home/dri/miner/freqtrade/.env/lib/python3.7/site-packages/joblib/_parallel_backends.py", line 521, in wrap_future_result
    return future.result(timeout=timeout)
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 435, in result
    return self.__get_result()
  File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
joblib.externals.loky.process_executor.TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker. The exit codes of the workers are {SIGKILL(-9)}


How i can debug this leak ?
It happens every time after some hour of work....
My linux kills the process after some time.


Discussion:
I am using latest git on a gentoo server with 8 threads and 16GB memory
is it really out of memory or just killed by some system service due to some other reasons?

you can monitor free memory with cacti, for example, see: https://cacti.net


It takes about 6-7 hours or more until this happens. Linux always kills the process if it eats all the memory. When i start it constantly eats eats memory until it is over then kernel kicks in with this:
```
[278069.827743] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/,task=python3.7,pid=12563,uid=1000
[278069.827749] Out of memory: Killed process 12563 (python3.7) total-vm:3148176kB, anon-rss:2363376kB, file-rss:0kB, shmem-rss:4kB
[278069.941328] oom_reaper: reaped process 12563 (python3.7), now anon-rss:0kB, file-rss:0kB, shmem-rss:4kB
```
It eats over 13GB of RAM for this time.
This is how i run it:
freqtrade -c config_kraken.json hyperopt -e 10000 --spaces all

I can duplicate the problem everytime just needs some hours.
If you guide me how i can help i will do it.



I can reproduce it.

But seems it's not a leak -- that's how optimizer works. It keeps the list of the optimization models that correspond to all previous epochs... And use shared memory segments for this to communicate with instances in the worker processes...

Should compare memory consumption with the old versions, dated by February/March...

Actually, you don't need to run hyperopt for thousands of epochs...

Yes maybe this is the way how it works but then you should put epoch limits or somehow clean memory ? I thought the more epoch the more good results you will have.....
As I said many times at Slack, hyperopt uses bayesian search, rather than grid optimization. It quickly descends to a local minimum of the objective function. Further epochs won't even find you some remarkably better points in the hyperspace of the optimization parameters or even won't find better points at all. Instead of running hyperopt one time for 10000 epochs it's better to run it 10 times for 1000 epochs -- it will find you 10 different local minimums you will be able to choose manually from later using other criteria. I personally never run hyperopting for more than 500-1000 epochs per run, except for testing/development purposes. It simply is not needed/useless.

What I can do here is to dump results into the pickling file every, say, 100 epochs so that they won't be lost.

Ok i can try testing like this:
freqtrade -c config.json hyperopt -e 100 --spaces all
Then when it finishes i will start again with
freqtrade -c config.json hyperopt --continue
Am i right ?
no
--continue is broken and never worked

better run 10 times
```
freqtrade -c config.json hyperopt -e 1000 --spaces all
```
and choose manually (or automatically on some criteria, best profit, for example) one of those 10 best results.
Ok i will run like that. I am going to close the ticket....
> --continue is broken and never worked

haven't seen any issues nor incentives towards fixing this ... what exactly is broken about it?
well, it simply does not use previous points.
How should it work: previously calculated points should be loaded into optimizer, the path should be 'replayed', and the optimizer should continue as if it was simply interrupted, from the same position.
So, ideally, if I run `hyperopt -e 100`, `hyperopt -e 100 --continue`, the result should be equal to `hyperopt -e 200`.
Same if hyperopt was interrupted by CtrlC or dumped core somewhere in the middle -- if restarted with `--continue`, it should continue optimization path starting with the last point which is in the results pickling file.

How it's implemented: the points are loaded to the hyperopt class but are not passed to the optimizer. The only usage of the results is the following: the minimum of the objective function in the loaded previous results is used as the current minimum and if one of the first epochs gives lesser value, it's printed as bold. That's all. The optimizer always start the epochs from scratches. You can see it with `--print-all --random-state N` and then `--print-all --random-state N --continue` -- in the second case the optimizer will start from the same set of points. It simply does not know anything about those previous results.

It's not that you broke it by introducing the `--continue` option -- that was the default behavior in the previous versions of freqtrade. And this has never been working as it should.

I'm not forcing it since I actually think that people rarely need the `--continue` functionality. It has more limitations, you know...

Heh, I started hyperopt on the server for 10000 epochs and went away for the day... It's now irresponsible, seems to be in deep swapping... 🤣 
Previous time it was killed with the same message as above. Should wait for couple of nights, I guess...

Yes at some point it will kill the process do not panic my server never stuck from that.
I don't panic, it was a joke (about couple of nights).
Hard reset rulez anyway

![memory-profile](https://user-images.githubusercontent.com/47309513/65374088-81a43580-dc8e-11e9-9edc-b97e84ba0dbc.png)

(it's not for 1000 epochs yet, interrupted after 200)

```
pip install memory_profiler
# run the profiler to record the memory usage
# sample 0.1s by default
mprof run --include-children freqtrade --options
# plot the recorded memory usage
mprof plot --output memory-profile.png
```
MemoryError at epoch 948:

![memory-profile](https://user-images.githubusercontent.com/47309513/65377474-19b51580-dcb5-11e9-99fe-fedf06b5f26e.png)

Just look at the overhead (time needed for epochs):

![65377474-19b51580-dcb5-11e9-99fe-fedf06b5f26e](https://user-images.githubusercontent.com/47309513/65380644-3a4a9300-dce8-11e9-8389-b6021328d25c.png)

The reason is the following: after second respawn of the worker processes, Loky/joblib starts to respawn them frequently. So we see the memory whipsaw... Since the size of the main process grows a bit, the re-forked workers also take more and more memory... Which finally results in the MemoryError... I have not found any leaks in the freqtrade code yet.

How to see the moments: When Loky re-forks the workers, you'll see the skilearn deprecation warnings in the output.

what is changing after epoch 380? 

it seems quite constant until then and suddenly seems to start becoming "fluky" ... ? 
i could understand if it's constantly like this ... but this looks like "something" is changing after 380 epochs (probably a random number, may be different in different trials?)

> what is changing after epoch 380?

Did not find the reason yet, Loky stops and respawns workers... Pickling is okay -- it only passes a small dict to the worker, shmem is not used.

Workers get terminated and respawned frequently after second respawn...

Our code seems be okay (looked at it with `mprof`, `objgraph`, `heapy`, in both workers and main process...

It's not always after 240/380 epochs, but depends on time, as I guess (Loky respawns them by timeout, probably)

Results with
```
with parallel_backend('multiprocessing'):
    ...
```

![memory-profile](https://user-images.githubusercontent.com/47309513/65386648-313cde80-dd47-11e9-96b7-6ae496980e21.png)

with multiprocessing debug switched on:

```
   236/1000:     20 trades. Avg profit -0.71%. Total profit -142.87529301 USDT ( -14.27Σ%). Avg duration 2232.0 mins. Objective: 1.04758
[INFO/LokyProcess-4] Memory leak detected: shutting down worker
[INFO/LokyProcess-4] process shutting down
[DEBUG/LokyProcess-4] running all "atexit" finalizers with priority >= 0
[DEBUG/LokyProcess-4] running the remaining "atexit" finalizers
[INFO/LokyProcess-4] process exiting with exitcode 0
[INFO/LokyProcess-2] Memory leak detected: shutting down worker
[INFO/LokyProcess-2] process shutting down
[DEBUG/LokyProcess-2] running all "atexit" finalizers with priority >= 0
[DEBUG/LokyProcess-2] running the remaining "atexit" finalizers
[INFO/LokyProcess-2] process exiting with exitcode 0
   237/1000:      6 trades. Avg profit -1.43%. Total profit -85.82721071 USDT (  -8.57Σ%). Avg duration 3320.0 mins. Objective: 1.02858
   238/1000:      0 trades. Avg profit   nan%. Total profit  0.00000000 USDT (   0.00Σ%). Avg duration   nan mins. Objective: 100000.00000
   239/1000:      5 trades. Avg profit -1.88%. Total profit -93.92421241 USDT (  -9.38Σ%). Avg duration 4000.0 mins. Objective: 1.03128
   240/1000:      4 trades. Avg profit -1.23%. Total profit -49.31936974 USDT (  -4.93Σ%). Avg duration 323.8 mins. Objective: 1.01642
```

-- it does not mean 'memory leak' specifically. Loky contains protection against memory leaks, measures periodically the memory used by the processes, and compares it against a hardcoded thrashold value, which is hardcoded as `1e8` = 100MB. So if the application creates lists, dicts or objects for 100MB, uses them and then gracefully frees, the memory leak can also be detected.

https://github.com/joblib/joblib/blob/470a6372dbae826b2f6107a77f6457e956249fef/joblib/externals/loky/process_executor.py#L124
> what is changing after epoch 380?

The application creates and freeze larger arrays or lists or something in its work (you can see above on the memory usage graphs -- the line is wider and wider with time), so after this point (it depends on the system, hyperopt and parameters used, in my debugging environment I obtain these epochs) Loky starts detecting what it calls memory leak after almost every iteration so it stops workers and restarts them...

What helps:
* using `multiprocessor` parallel_backend instead of Loky; this is not recommended by the creators of joblib, though
* `pip uninstall psutil` -- in this case, Loky falls back to that code that does not use psutil and does not stop the workers even if it detects leaks
* increase `joblib.externals.loky.process_executor._MAX_MEMORY_LEAK_SIZE = int(5e8)` (there are many joblibs, in skopt/sklearn and standalone...)
* find the reason why it uses so much memory. Objgraph says everything is okay in the bot, we have increasing amount of dicts, lists and strs at each epoch, but they only take 30+20 MB after 1000 epochs...

can it be the backtesting data itself? it's a list (or dataframe?) of a few 1000 ohlcv candles which is unpickled within each job task?

it's not something we can circumvent though as this data is necessary for backtesting to work.

> increase joblib.externals.loky.process_executor._MAX_MEMORY_LEAK_SIZE = int(5e8) (there are many joblibs, in skopt/sklearn and standalone...)

we should only be using standalone (direct imports -  `from joblib import ...`) ... unless skopt does call it again within itself from somewhere else? which could cause this problem, since skopt uses a older version without wrappable (fell into this trap a while ago).



My Holy Cow, ask me how! 

(1000 epochs as above,,,):

![memory-profile](https://user-images.githubusercontent.com/47309513/65403707-c38fc180-dddd-11e9-8b39-59394f3c13aa.png)

(And note the time in seconds, compare with previous results above... :smile: )
> My Holy Cow, ask me how!

i'll not need to hopefully, as i suspect that we can look at the way for this improvement in the next PR... 🚀 
Yeah, felt asleep waiting for results of 1000 epochs session...

The problem here is that skopt seems to lost support, it's not updated for more than half an year...
So I embedded workaround into our main hyperopt loop...


Resolving patch:
diff --git a/freqtrade/optimize/hyperopt.py b/freqtrade/optimize/hyperopt.py
--- a/freqtrade/optimize/hyperopt.py
+++ b/freqtrade/optimize/hyperopt.py
@@ -36,6 +36,11 @@
 
 
 INITIAL_POINTS = 30
+
+# Keep no more than 2*SKOPT_MODELS_MAX_NUM models
+# in the skopt models list
+SKOPT_MODELS_MAX_NUM = 10
+
 MAX_LOSS = 100000  # just a big enough number to be bad result in loss optimization
 
 
@@ -255,12 +260,13 @@ def hyperopt_space(self, space: Optional[str] = None) -> List[Dimension]:
             spaces += self.custom_hyperopt.stoploss_space()
         return spaces
 
-    def generate_optimizer(self, _params: Dict) -> Dict:
+    def generate_optimizer(self, _params: Dict, iteration=None) -> Dict:
         """
         Used Optimize function. Called once per epoch to optimize whatever is configured.
         Keep this function as optimized as possible!
         """
         params = self.get_args(_params)
+
         if self.has_space('roi'):
             self.backtesting.strategy.minimal_roi = \
                     self.custom_hyperopt.generate_roi_table(params)
@@ -342,9 +348,26 @@ def get_optimizer(self, dimensions, cpu_count) -> Optimizer:
             random_state=self.config.get('hyperopt_random_state', None)
         )
 
-    def run_optimizer_parallel(self, parallel, asked) -> List:
+    def fix_optimizer_models_list(self):
+        """
+        WORKAROUND: Since skopt is not actively supported, this resolves problems with skopt
+        memory usage, see also: https://github.com/scikit-optimize/scikit-optimize/pull/746
+
+        This may cease working when skopt updates if implementation of this intrinsic
+        part changes.
+        """
+        n = len(self.opt.models) - SKOPT_MODELS_MAX_NUM
+        # Keep no more than 2*SKOPT_MODELS_MAX_NUM models in the skopt models list,
+        # remove the old ones. These are actually of no use, the current model
+        # from the estimator is the only one used in the skopt optimizer.
+        # Freqtrade code also does not inspect details of the models.
+        if n >= SKOPT_MODELS_MAX_NUM:
+            logger.debug(f"Fixing skopt models list, removing {n} old items...")
+            del self.opt.models[0:n]
+
+    def run_optimizer_parallel(self, parallel, asked, i) -> List:
         return parallel(delayed(
-                        wrap_non_picklable_objects(self.generate_optimizer))(v) for v in asked)
+                        wrap_non_picklable_objects(self.generate_optimizer))(v, i) for v in asked)
 
     def load_previous_results(self):
         """ read trials file if we have one """
@@ -407,8 +430,9 @@ def start(self) -> None:
                 EVALS = max(self.total_epochs // jobs, 1)
                 for i in range(EVALS):
                     asked = self.opt.ask(n_points=jobs)
-                    f_val = self.run_optimizer_parallel(parallel, asked)
+                    f_val = self.run_optimizer_parallel(parallel, asked, i)
                     self.opt.tell(asked, [v['loss'] for v in f_val])
+                    self.fix_optimizer_models_list()
                     for j in range(jobs):
                         current = i * jobs + j
                         val = f_val[j]


Test patch: