opened
ramnath-k
username_0
### What related GitHub issues or StackOverflow threads have you found by searching the web for your problem?
Closest I found was https://github.com/tensorflow/tensorflow/issues/4767
But I don't intend to change the datatypes  of any variables...

### Environment info
Operating System: Ubuntu 14.04

Installed version of CUDA and cuDNN: CUDA 7.5 and CUDNN 4.0.7
(please attach the output of `ls -l /path/to/cuda/lib/libcud*`):
-rw-r--r-- 1 root root 189170 Aug 25 02:29 /usr/local/cuda/lib/libcudadevrt.a
lrwxrwxrwx 1 root root     16 Aug 25 02:29 /usr/local/cuda/lib/libcudart.so -> libcudart.so.7.5
lrwxrwxrwx 1 root root     19 Aug 25 02:29 /usr/local/cuda/lib/libcudart.so.7.5 -> libcudart.so.7.5.18
-rwxr-xr-x 1 root root 311596 Aug 25 02:29 /usr/local/cuda/lib/libcudart.so.7.5.18
-rw-r--r-- 1 root root 558020 Aug 25 02:29 /usr/local/cuda/lib/libcudart_static.a

CUDNN libs are in /cuda/lib64/ (output of `ls -l /path/to/cuda/lib64/libcud*`):
-rw-r--r-- 1 root root   322936 Aug 25 02:29 /usr/local/cuda/lib64/libcudadevrt.a
lrwxrwxrwx 1 root root       16 Aug 25 02:29 /usr/local/cuda/lib64/libcudart.so -> libcudart.so.7.5
lrwxrwxrwx 1 root root       19 Aug 25 02:29 /usr/local/cuda/lib64/libcudart.so.7.5 -> libcudart.so.7.5.18
-rwxr-xr-x 1 root root   383336 Aug 25 02:29 /usr/local/cuda/lib64/libcudart.so.7.5.18
-rw-r--r-- 1 root root   720192 Aug 25 02:29 /usr/local/cuda/lib64/libcudart_static.a
-rwxr-xr-x 1 root root 61453024 Aug 25 02:36 /usr/local/cuda/lib64/libcudnn.so
-rwxr-xr-x 1 root root 61453024 Aug 25 02:36 /usr/local/cuda/lib64/libcudnn.so.4
-rwxr-xr-x 1 root root 61453024 Aug 25 02:36 /usr/local/cuda/lib64/libcudnn.so.4.0.7
-rw-r--r-- 1 root root 62025862 Aug 25 02:36 /usr/local/cuda/lib64/libcudnn_static.a

But **I have installed Tensorflow CPU only version in the virtualenv I am testing this code on**, hence it might not be using CUDA at all as far as this issue is concerned (I am debugging an issue that occurred on a EC2 machine which has no GPU, by reproducing it in my local system)
 
If installed from binary pip package, provide:

1. A link to the pip package you installed: https://storage.googleapis.com/tensorflow/linux/cpu/tensorflow-0.11.0rc2-cp27-none-linux_x86_64.whl

2. The output from `python -c "import tensorflow; print(tensorflow.__version__)"`.0.11.0rc2

### If possible, provide a minimal reproducible example (We usually don't have time to read hundreds of lines of your code)
I have saved a minimal working code which reproduces this issue at [gist](https://gist.github.com/username_0/d299964fead58c0d1e0df9c2190a4f91)

To reproduce the issue:
1. Call saver.save in the main thread and checkpoint at least one tf.Variable
2. loading this checkpoint from main thread and evaluating the variable works fine
3. if I now launch a subprocess and try to load the checkpoint in that, Tensorflow hangs at sess.run(tf.initialize_all_variables())

### What other attempted solutions have you tried?
I tried putting a container with names suffixed by the subprocess pid but it didn't help. I also tried a basic version of create_local_server and it didn't work too

### Logs or other output that would be helpful
(If logs are large, please upload as attachment or provide link).
I don't see any errors in the console when the code hangs.
Tensorflow hangs when initializing variables in a multi process setting
issue
created
mrry
username_1
It looks like the issue here is that Python 2.7 unconditionally uses `os.fork()` in to implement `multiprocessing.Process.start()`, but the TensorFlow runtime is not fork-safe (for one thing, because it creates various internal threads and initialized static members at startup). Since in your example the parent process invokes the TensorFlow runtime, the child will be in an unspecified state when it attempts to do the first `sess.run()` call. (When I ran your code, it was blocking on a condition variable, waiting for constant propagation to finish, for example.)

As far as I can tell, there is no way to make the Python 2.7 multiprocessing module do the "right thing" here. If you upgrade to Python 3.4, you can use [`multiprocessing.set_start_method('spawn')`](https://docs.python.org/3/library/multiprocessing.html#multiprocessing.set_start_method) to avoid the issues over fork-safety. Alternatively, you could structure your program so that the `import tensorflow as tf` statement only runs in child processes.
comment
created
ramnath-k
username_0
Thanks @username_1 
I tried out your suggestion to move the tensorflow code over to child processes. 
First up I moved the saving and restoring code to another module as in this [basic_saver_restore.py](https://gist.github.com/username_0/2990b46b84278e5e24919beb44968a00)
Then I called the saver and restore as before using multiprocessing as in [multi_process_saver_restore.py](https://gist.github.com/username_0/fa25696f3f1def6ba32855f612353914)
In basic_saver_restore, I put a module level variable server equal to create_local_server (which I assume creates a child process to run the session right?). 
This seems to fix the hang I was getting. 
Will this approach always work? Or is it that I am only fixing some issues but not all with this approach?

Moving to Python 3 is something I will attempt later.

My actual hang occurs when a Hive launches a python user defined function which calls a restore tensorflow model function. Do you think this approach of having a module level create_local_server fix such an issue also?
comment
created
sherrym
username_2
Hi @username_0 ,

It appears to have hung because it's waiting for the main thread (or process) to yield the global interpreter lock. Make your processes daemon processes and it should work.

  for i in range(num_procs):
      p = Process(target=subprocess, args=(i,))
      p.daemon = True            <<<<< Add this line.
      p.start() # process fails to get past initialize all variables
      procs.append(p)

I just tried it and it worked for me. I also made some simplifications:

def add_model():
  a = tf.Variable(2, name='a')
  b = tf.Variable(5, name='b')
  c = tf.mul(a, b, name='c')
  return a, b, c

def save_session(unused_arg=None):
  with tf.Session(graph=tf.Graph()) as sess:
    a, b, c = add_model()
    saver = tf.train.Saver(
        tf.global_variables(),
        max_to_keep=1)
    init = [
        tf.global_variables_initializer(),
        tf.local_variables_initializer()]
    sess.run(init)
    e = tf.assign(a, 3, name='e')
    f = tf.assign(b, 4, name='f')
    sess.run([e, f])
    val = sess.run(c)
    print('val=', val)
    checkpoint_dir = './debug'
    if not os.path.exists(checkpoint_dir):
      os.makedirs(checkpoint_dir)
    checkpoint_prefix = os.path.join(checkpoint_dir, 'debug')
    path = saver.save(
        sess, checkpoint_prefix)
    print('save session complete')

def restore_session(unused_arg=None):
  graph = tf.Graph()
  with tf.Session(graph=graph) as sess:
    pid = os.getpid()
    container_name = 'worker{}'.format(pid)
    print('container:{}'.format(container_name))
    with graph.container(container_name):
      a, b, c = add_model()
      saver = tf.train.Saver(tf.global_variables())
      print('add model complete')
      init = [
          tf.global_variables_initializer(),
          tf.local_variables_initializer()]
      sess.run(init)
      print('init model complete')
      graph.finalize()
      model_checkpoint ='debug/debug'
      saver.restore(sess, model_checkpoint)
      val = sess.run(c)
      print('val=', val)


def subprocess(i):
  print('inside subprocess {}'.format(i))
  restore_session()
  print('exiting subprocess {}'.format(i))


def main(unused_argv):
  save_p = Process(target=save_session, args=(1,))
  save_p.start()
  save_p.join()
  restore_p = Process(target=restore_session, args=(1,))
  restore_p.start()
  restore_p.join()
  procs = []
  num_procs = 3
  for i in range(num_procs):
      p = Process(target=subprocess, args=(i,))
      p.daemon = True
      p.start() # process fails to get past initialize all variables
      procs.append(p)
  for p in procs:
      p.join()

if __name__ == "__main__":
  tf.app.run()


Please let me know if this works for you. Thanks.

Sherry
comment
created
ramnath-k
username_0
Thanks for the update @username_2
Yes moving everything to child processes makes it work. But to clarify, in the modified code you have provided I don't even need to set daemon=True as no Tensorflow session has been created in the main process.
It hangs only if I start a session in the main process and then fork a child process.

Scenario is something like, only after the saver completes it must launch multiple workers which process the saved checkpoint.
comment
closed
sherrym
username_2
issue
created
hiemal
username_3
I use keras as wrapper. Moving all the imports into subprocess solves the issue.
comment
created
nottombrown
username_4
@username_1's Python3 solution worked for me. Thanks!

```python
import multiprocessing

import tensorflow as tf

def f(x):
    session = tf.Session()
    a = tf.Variable(x, name='a')
    b = tf.Variable(100, name='b')
    c = tf.multiply(a, b, name='c')
    session.run(tf.global_variables_initializer())

    out = session.run(c)
    print("OK: %s" % out)

if __name__ == '__main__':
    multiprocessing.set_start_method('spawn')  # Comment me out to hang
    f(0)
    multiprocessing.Pool().map(f, range(10))
```
comment
created
markroxor
username_5
@username_1 Your solution raises -
```python3
_pickle.PicklingError: Can't pickle <class 'module'>: attribute lookup module on builtins failed
```
I can provide more information if required.
comment
created
zaccharieramzi
username_6
I am trying to solve https://github.com/tensorflow/tensorflow/issues/34456 , so I tried the main solution provided here (`multiprocessing.set_start_method('spawn')`), but I got the following error: `TypeError: can't pickle _thread.lock objects`.

This is because [tf is initializing a pool with a queue](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/utils/data_utils.py#L821-L823) which is [apparently not doable](https://stackoverflow.com/a/7865512/4332585).
comment