Repository: Theano/Theano

Issue:
Memory gradually increasing until crash while training simple MLP on CPU
I am running this [tutorial](http://deeplearning.net/tutorial/mlp.html) that is training a simple MLP on MNIST. I run on CPU. While monitoring my computer activity in the training loop, I can see that the memory usage is steadily increasing, until it reaches 100% and everything crashes (need to reboot).

I am on Ubuntu 16.04 and using the bleeding edge version of Theano.
There has been a similar [post](https://github.com/Theano/Theano/issues/4101) but the problem seemed to arise exclusively on GPU.

Is anyone experiencing the same issue on CPU?


Discussion:
I also encountered this issue, except I am on Windows 7 and am using stable 0.9.0 with [Keras](https://keras.io/) 1.2.2 and Python 2.7. Memory usage of the python.exe process increases until the system is unusable. Reverting to Theano 0.8.2 resolves the issue. 
Thanks for the reply! How exactly can you revert to an older version of Theano? Do you need to uninstall Theano first? and how to specify the version with pip install?
Reverting is pretty easy with pip. Just use "pip install theano==[version]" to install a specific version. In my case, "pip install theano==0.8.2." It should automatically uninstall the current version.

For another reference to this issue, users of Keras have also [encountered the leak and apparently determined that the cause is in Theano 0.9.0](https://github.com/fchollet/keras/issues/5935).
Any specific Theano flags?
Do you have a vague idea of the rate of growth of memory usage? i.e., how long until it fills your memory, and how large is it? I'm trying to reproduce locally, and I'm not sure how long I should wait for the effects to manifest.

If you can clone the Theano repository from GitHub, and use `git bisect` to find the first "bad" commit between tags for the 0.8.0 and 0.9.0 releases, that would be awesome.
I'm running with Python 2.7, numpy 1.11, floatX=float32, and after more than 30 minutes, `top` still reports `328512` (kB) in "RES" memory, which does not change.
Do you use Python 3.6? I have see problems report related to numpy on
python 3.6. Make sure to update numpy to the last release if on python 3.6.
Otherwise, downgrade to 3.5. We didn't tested 3.6 yet, so don't support it
officially. Personnaly, I would not expect problem, but I got enough report
on it that I don't recommand to use numpy or Theano on python 3.6 for now.

On Fri, Apr 7, 2017 at 7:17 PM Pascal Lamblin <notifications@github.com>
wrote:

> I'm running with Python 2.7, numpy 1.11, floatX=float32, and after more
> than 30 minutes, top still reports 328512 (kB) in "RES" memory, which
> does not change.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/Theano/Theano/issues/5810#issuecomment-292673669>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AALC-2zqk3Lxqb2bpLL-qLKtAO3I1PBbks5rtsP1gaJpZM4MzsVn>
> .
>

I have 8GB of RAM and it gets from 3 to 8GB in a matter of seconds right after launching the script mentioned above. I am using python 2.7 and numpy 1.12.1.

For now I have downgraded to Theano 0.8.2 as @abgcsc has suggested and the problem is gone.


Which Python version?

On Fri, Apr 7, 2017 at 10:54 PM Julien <notifications@github.com> wrote:

> I have 8GB of RAM and it gets from 3 to 8GB in a matter of seconds right
> after launching the script mentioned above. I am using python 2.7 and numpy
> 1.12.1.
>
> For now I have downgraded to Theano 0.8.2 as @abgcsc
> <https://github.com/abgcsc> has suggested and the problem is gone.
>
> —
> You are receiving this because you commented.
>
>
> Reply to this email directly, view it on GitHub
> <https://github.com/Theano/Theano/issues/5810#issuecomment-292690059>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AALC-0VX1jnxNkxQa9LxpBaql__n4NUoks5rtvbfgaJpZM4MzsVn>
> .
>

I am also using Python 2.7 and Numpy 1.12.1 with 16 GB of RAM. The leak is quite noticeable at 200-300MB per second on my machine when running the MLP tutorial.

Here is the output of git bisect:

da800d748bcd0c79d5bb1ad71b7eedb9a306afd6 is the first bad commit
commit da800d748bcd0c79d5bb1ad71b7eedb9a306afd6
Author: slefrancois <simon.lefrancois@umontreal.ca>
Date:   Tue Aug 30 09:55:57 2016 -0400

    port missing doc PRs to master

:040000 040000 e0eea7dfedb49fce15dc4b56709582e35c8c52cd 3a30f29020fa7543dcfa6076 5dfc812e4c85523a M      doc
:040000 040000 4209ad9c6eb233767ad63649c56efe1e5ada0ba5 16256d18b363e0377f62e3f3 37d624766874d74d M      theano
I am experiencing this issue on Windows 10, Theano 0.9.0, Python 2.7.11, Numpy 1.12.1

Reverting to Theano 0.8.2 has resolved the issue, but unfortunately some libraries I use depend on Theano 0.9.0 syntax
@abgcsc This commit touches only documentation, and an unused configuration option. Could you try another bisect, but calling `theano-cache purge` at each new checkout?
I will try with numpy 1.12.1 since it seems to be a common factor.
@lamblin Yeah, I thought the commit looked fairly benign, but I didn't want to make any assumptions. Here are the results of another bisect with a cache purge at each iteration:

4185ef0c4475086f03d5e15419a7085c87b412a6 is the first bad commit
commit 4185ef0c4475086f03d5e15419a7085c87b412a6
Author: notoraptor <stevenbocco@gmail.com>
Date:   Wed Oct 26 16:10:42 2016 -0400

    New update.
    Code-comment removed from test_corr.
    Code factorized in alt_gemm_template.c

    All tests passed.

:040000 040000 f2a8681188571e260167a6613509bd88a78d50ac b6930bca1919e43b36628ddf
d14420ee4d4084da M      theano
Oh, it would make more sense, then.
This code falls back on NumPy's C API for dot products when we cannot directly link with a BLAS, and should not be selected when numpy has been installed with Anaconda (which provides MKL bindings), which may be why I did not manage to reproduce it.
For the moment, you could try to either:
- install numpy and other requirements through Anaconda, as well as `mkl`
- install a version of BLAS through your OS, and add the flags required to link with it in the config variable `blas.ldflags`

There's probably a missing decref somewhere in the code, I'll have a look.

Resolving patch:
diff --git a/theano/tensor/blas_headers.py b/theano/tensor/blas_headers.py
--- a/theano/tensor/blas_headers.py
+++ b/theano/tensor/blas_headers.py
@@ -1032,7 +1032,7 @@ def openblas_threads_text():
 
 def blas_header_version():
     # Version for the base header
-    version = (1,)
+    version = (2,)
     if detect_macos_sdot_bug():
         if detect_macos_sdot_bug.fix_works:
             # Version with fix


Test patch: