Repository: Theano/Theano

Issue:
Getting File Exists Error when running Theano on shared CPU
Theano Version: '1.0.2'

I get the following error:

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/rishi_gurnani/localpy/Python-3.5.5/Lib/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/rishi_gurnani/localpy/Python-3.5.5/Lib/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/rishi_gurnani/Code/Mine/CV_sc.py", line 142, in wrapper
    num_epochs_fold=num_epochs_fold)
  File "/home/rishi_gurnani/Code/Mine/CV_sc.py", line 36, in CV
    y[i] = x_2(x_1a(inputs[i]))
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/engine/base_layer.py", line 432, in __call__
    self.build(input_shapes[0])
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/layers/core.py", line 872, in build
    constraint=self.kernel_constraint)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/engine/base_layer.py", line 249, in add_weight
    weight = K.variable(initializer(shape),
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/initializers.py", line 218, in __call__
    dtype=dtype, seed=self.seed)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/backend/theano_backend.py", line 2504, in random_uniform
    return rng.uniform(shape, low=minval, high=maxval, dtype=dtype)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/sandbox/rng_mrg.py", line 872, in uniform
    rstates = self.get_substream_rstates(nstreams, dtype)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/configparser.py", line 117, in res
    return f(*args, **kwargs)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/sandbox/rng_mrg.py", line 779, in get_substream_rstates
    multMatVect(rval[0], A1p72, M1, A2p72, M2)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/sandbox/rng_mrg.py", line 62, in multMatVect
    [A_sym, s_sym, m_sym, A2_sym, s2_sym, m2_sym], o, profile=False)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/compile/function.py", line 317, in function
    output_keys=output_keys)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/compile/pfunc.py", line 486, in pfunc
    output_keys=output_keys)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/compile/function_module.py", line 1841, in orig_function
    fn = m.create(defaults)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/compile/function_module.py", line 1715, in create
    input_storage=input_storage_lists, storage_map=storage_map)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/link.py", line 699, in make_thunk
    storage_map=storage_map)[:3]
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/vm.py", line 1091, in make_all
    impl=impl))
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/op.py", line 955, in make_thunk
    no_recycling)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/op.py", line 858, in make_c_thunk
    output_storage=node_output_storage)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/cc.py", line 1217, in make_thunk
    keep_lock=keep_lock)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/cc.py", line 1157, in __compile__
    keep_lock=keep_lock)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/cc.py", line 1620, in cthunk_factory
    key=key, lnk=self, keep_lock=keep_lock)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/cmodule.py", line 1151, in module_from_key
    with compilelock.lock_ctx(keep_lock=keep_lock):
  File "/home/rishi_gurnani/localpy/Python-3.5.5/Lib/contextlib.py", line 59, in __enter__
    return next(self.gen)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/compilelock.py", line 39, in lock_ctx
    get_lock(lock_dir=lock_dir, **kw)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/compilelock.py", line 85, in _get_lock
    lock(get_lock.lock_dir, **kw)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/compilelock.py", line 275, in lock
    os.mkdir(tmp_dir)
FileExistsError: [Errno 17] File exists: '/home/rishi_gurnani/.theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-centos-6.4-Final-x86_64-3.5.5-64/lock_dir'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/rishi_gurnani/jobs/tests/opt_c/Jun11_1/jobs.py", line 34, in <module>
    (optimal_coeffs, dump_folder_path, num_splits, num_epochs_final, num_rdf, num_adf, distinct_singlet_index, distinct_pairs_index, r_cutoff) = optimize(partial_path_to_xyz_files, partial_dump_path,r_cutoff, num_total, num_rdf, num_adf)
  File "/home/rishi_gurnani/Code/Mine/CV_sc.py", line 220, in optimize
    resfile=dump_folder_path+'%s_%s_%s_%s_hyperopt.csv' % (num_adf, num_rdf, n, m))  # text file where results will be saved
  File "/home/rishi_gurnani/Code/Mine/blackbox.py", line 289, in searchr
    points[batch*i:batch*(i+1), -1] = list(e.map(f, list(map(cubetobox, points[batch*i:batch*(i+1), 0:-1]))))
  File "/home/rishi_gurnani/localpy/Python-3.5.5/Lib/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/rishi_gurnani/localpy/Python-3.5.5/Lib/multiprocessing/pool.py", line 644, in get
    raise self._value
FileExistsError: [Errno The following error happened while compiling the node] DotModulo(A, s, m, A2, s2, m2): '\n' -> 'File exists'

So far to remedy this I have tried:

1. Deleting my ~/.theano folder before running the code

2. Adding to my ~/.theanorc file:
[global]
config.compile.timeout = 1000

3. Adding to my ~/.theanorc file:
[global]
config.compile.timeout = 5000

The third attempt above gave me a new error. Not sure if this is progress or not.


Discussion:
Can you try with this diff:

```diff
diff --git a/theano/gof/compilelock.py b/theano/gof/compilelock.py
index d5e2d8230..d450f3273 100644
--- a/theano/gof/compilelock.py
+++ b/theano/gof/compilelock.py
@@ -273,7 +273,7 @@ def lock(tmp_dir, timeout=notset, min_wait=None, max_wait=None, verbosity=1):
 
             try:
                 os.mkdir(tmp_dir)
-            except OSError:
+            except (OSError, FileExistsError):
                 # Error while creating the directory: someone else
                 # must have tried at the exact same time.
                 nb_error += 1
```
@abergeron 
I did as you suggested and still got a FileExistsError, albeit from a different place. See below.

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/lazylinker_c.py", line 81, in <module>
    actual_version, force_compile, _need_reload))
ImportError: Version check of the existing lazylinker compiled file. Looking for version 0.211, but found None. Extra debug information: force_compile=False, _need_reload=True

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/vm.py", line 674, in <module>
    from . import lazylinker_c
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/lazylinker_c.py", line 83, in <module>
    get_lock()
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/compilelock.py", line 85, in _get_lock
    lock(get_lock.lock_dir, **kw)
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/compilelock.py", line 275, in lock
    os.mkdir(tmp_dir)
FileExistsError: [Errno 17] File exists: '/home/rishi_gurnani/.theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-centos-6.4-Final-x86_64-3.5.5-64/lock_dir'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/rishi_gurnani/localpy/Python-3.5.5/Lib/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/home/rishi_gurnani/localpy/Python-3.5.5/Lib/multiprocessing/pool.py", line 44, in mapstar
    return list(map(*args))
  File "/home/rishi_gurnani/Code/Mine/CV_sc.py", line 142, in wrapper
    num_epochs_fold=num_epochs_fold)
  File "/home/rishi_gurnani/Code/Mine/CV_sc.py", line 15, in CV
    from keras.layers import Input, Dense
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/__init__.py", line 3, in <module>
    from . import utils
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/utils/__init__.py", line 6, in <module>
    from . import conv_utils
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/utils/conv_utils.py", line 9, in <module>
    from .. import backend as K
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/backend/__init__.py", line 84, in <module>
    from .theano_backend import *
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/keras/backend/theano_backend.py", line 7, in <module>
    import theano
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/__init__.py", line 110, in <module>
    from theano.compile import (
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/compile/__init__.py", line 12, in <module>
    from theano.compile.mode import *
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/compile/mode.py", line 11, in <module>
    import theano.gof.vm
  File "/home/rishi_gurnani/localpy/lib/python3.5/site-packages/theano/gof/vm.py", line 689, in <module>
    if x.fullname == 'linker'][0].default.startswith('cvm'), e
AssertionError: [Errno 17] File exists: '/home/rishi_gurnani/.theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-centos-6.4-Final-x86_64-3.5.5-64/lock_dir'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/rishi_gurnani/jobs/tests/opt_c/Jun11_1/jobs.py", line 34, in <module>
    (optimal_coeffs, dump_folder_path, num_splits, num_epochs_final, num_rdf, num_adf, distinct_singlet_index, distinct_pairs_index, r_cutoff) = optimize(partial_path_to_xyz_files, partial_dump_path,r_cutoff, num_total, num_rdf, num_adf)
  File "/home/rishi_gurnani/Code/Mine/CV_sc.py", line 220, in optimize
    resfile=dump_folder_path+'%s_%s_%s_%s_hyperopt.csv' % (num_adf, num_rdf, n, m))  # text file where results will be saved
  File "/home/rishi_gurnani/Code/Mine/blackbox.py", line 289, in searchr
    points[batch*i:batch*(i+1), -1] = list(e.map(f, list(map(cubetobox, points[batch*i:batch*(i+1), 0:-1]))))
  File "/home/rishi_gurnani/localpy/Python-3.5.5/Lib/multiprocessing/pool.py", line 266, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/home/rishi_gurnani/localpy/Python-3.5.5/Lib/multiprocessing/pool.py", line 644, in get
    raise self._value
AssertionError: [Errno 17] File exists: '/home/rishi_gurnani/.theano/compiledir_Linux-2.6-el6.x86_64-x86_64-with-centos-6.4-Final-x86_64-3.5.5-64/lock_dir'
Maybe the above error is because in my initial debug attempt I make my script delete ~/.theano?
Do you have any Theano process running? If so, can you stop them. Delete you ~/.theano directory again? It is the first time I see this error.
@nouiz In my python script, before I use theano I automatically delete ~/.theano if it exists using shutil.rmtree() each time.
@nouiz @abergeron 

So I deleted my vim ~/.theanorc but am still using the diff that @abergeron suggested. The initial error message was reproduced. I think this means that, in /theano/gof/compilelock.py, nb_error >= 10?
It worked when I changed nb_error to 100
@nouiz `FileExistsError` is a new python 3 exception that `os.makedir()` can raise (and `open()` too). In python 2 it raises `OSError`.  Apparently nobody ever had lock contention on python 3 before.

@rishigurnani How many processes are you running?  You would have to be running a *lot* to have so much contention that you can lose the race 10 times in a row.
@abergeron Not sure how to determine # of processes, but I'm running 64 cores

Resolving patch:
diff --git a/theano/gof/compilelock.py b/theano/gof/compilelock.py
--- a/theano/gof/compilelock.py
+++ b/theano/gof/compilelock.py
@@ -7,6 +7,7 @@
 import socket  # only used for gethostname()
 import time
 import logging
+from six import PY3
 
 from contextlib import contextmanager
 
@@ -271,9 +272,14 @@ def lock(tmp_dir, timeout=notset, min_wait=None, max_wait=None, verbosity=1):
                 nb_wait += 1
                 time.sleep(random.uniform(min_wait, max_wait))
 
+            if PY3:
+                exception = FileExistsError  # noqa
+            else:
+                exception = OSError
+
             try:
                 os.mkdir(tmp_dir)
-            except OSError:
+            except exception:
                 # Error while creating the directory: someone else
                 # must have tried at the exact same time.
                 nb_error += 1


Test patch: