Repository: ipython/ipython

Issue:
err at self.root.makedirs() if file exists
Hi,

I am not a super expert in ipython, so this is kind of a question. I got this error when using ipython to send jobs to a queue system. (using ipython through ipython-cluster-helper)

```
2015-03-05 01:38:53.600 [IPEngineApp] Using existing profile dir: u'/scratch/wgs10/work/log/ipython'
2015-03-05 01:38:53.616 [IPEngineApp] Loading url_file u'/scratch/wgs10/work/log/ipython/security/ipcontroller-94bc1b33-37a9-4$
e2-a547-265845dacafb-engine.json'
2015-03-05 01:38:53.629 [IPEngineApp] Registering with controller at tcp://10.0.0.15:37783
ERROR:tornado.general:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 407, in _run_call$ack
    callback(*args, **kwargs)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrappe$    return fn(*args, **kwargs)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/parallel/engine/engine.py", line 146, in <l$
mbda>
    self.registrar.on_recv(lambda msg: self.complete_registration(msg, connect, maybe_tunnel))
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/parallel/engine/engine.py", line 230, in co$
plete_registration
    loop=loop, user_ns=self.user_ns, log=self.log)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/kernel/zmq/ipkernel.py", line 52, in __init$
_
    kernel      = self,
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/config/configurable.py", line 337, in insta$
ce
    inst = cls(*args, **kwargs)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 490, in __in
it__
    self.db = PickleShareDB(os.path.join(self.profile_dir.location, 'db'))
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/utils/pickleshare.py", line 57, in __init__
    self.root.makedirs()
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/external/path/_path.py", line 1062, in maked
irs
    os.makedirs(self, mode)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/home/ubuntu/.ipython/profile_default/db'
ERROR:tornado.general:Uncaught exception, closing connection.
Traceback (most recent call last):
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 433, in _handle_ev
ents
    self._handle_recv()
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 465, in _handle_re
cv
    self._run_callback(callback, msg)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/zmq/eventloop/zmqstream.py", line 407, in _run_callb
ack
    callback(*args, **kwargs)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/parallel/engine/engine.py", line 146, in <la
mbda>
    self.registrar.on_recv(lambda msg: self.complete_registration(msg, connect, maybe_tunnel))
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/parallel/engine/engine.py", line 230, in com
plete_registration
    loop=loop, user_ns=self.user_ns, log=self.log)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/kernel/zmq/ipkernel.py", line 52, in __init_
_
    kernel      = self,
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/config/configurable.py", line 337, in instan
ce
    inst = cls(*args, **kwargs)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 490, in __in
it__
    self.db = PickleShareDB(os.path.join(self.profile_dir.location, 'db'))
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/utils/pickleshare.py", line 57, in __init__
    self.root.makedirs()
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/site-packages/IPython/external/path/_path.py", line 1062, in maked
irs
    os.makedirs(self, mode)
  File "/home/ubuntu/install/bcbio-vm/anaconda/lib/python2.7/os.py", line 157, in makedirs
    mkdir(name, mode)
OSError: [Errno 17] File exists: '/home/ubuntu/.ipython/profile_default/db'
```

I think the problem is this line: 
https://github.com/ipython/ipython/blob/71256f5f771a542589ffe9780cba1dbea0bffafb/IPython/utils/pickleshare.py#L57

and is because, in this case, I was having 4 processes going at the same time,  that maybe one of them create the folder while the other went through the if because wasn't there at that precise moment, but it was when tried to created. 

I saw that `self.root.makedirs()` call  `os.makedirs()`. I think that this problem will disappear if is used: `self.root.makedirs_p()`.  or using  `os.makedirs(name, mode=0o777, exist_ok=True)` in the `Path class`. 

If i am not making any sense, sorry about it. It is the first time i have this error, and I am testing in a cluster in aws, so only I am the user, and all jobs and code execute at the same time... this is my guess.

feel free to close it, if this is not useful.



Discussion:
Ha, this is a classic mistake that hardly ever causes an actual problem - check if a file exists, and then immediately do something that fails if it's created/deleted since the check.

`.makedirs_p()` does look like the right fix. Do you want to make a PR?

You should also report it against the independent pickleshare package, which appears to have the same flaw: https://github.com/vivainio/pickleshare/blob/master/pickleshare.py#L61 . We're hoping to get rid of our bundled copy and depend on that in the future, but we're waiting for a new release before we do.

nice! will prepare the PR! i usually get the most unlikely errors :)

sent not from my computer

> On Mar 5, 2015, at 21:06, Thomas Kluyver notifications@github.com wrote:
> 
> Ha, this is a classic mistake that hardly ever causes an actual problem - check if a file exists, and then immediately do something that fails if it's created/deleted since the check.
> 
> .makedirs_p() does look like the right fix. Do you want to make a PR?
> 
> You should also report it against the independent pickleshare package, which appears to have the same flaw: https://github.com/vivainio/pickleshare/blob/master/pickleshare.py#L61 . We're hoping to get rid of our bundled copy and depend on that in the future, but we're waiting for a new release before we do.
> 
> —
> Reply to this email directly or view it on GitHub.


Resolving patch:
diff --git a/IPython/utils/pickleshare.py b/IPython/utils/pickleshare.py
--- a/IPython/utils/pickleshare.py
+++ b/IPython/utils/pickleshare.py
@@ -54,7 +54,7 @@ def __init__(self,root):
         """ Return a db object that will manage the specied directory"""
         self.root = Path(root).expanduser().abspath()
         if not self.root.isdir():
-            self.root.makedirs()
+            self.root.makedirs_p()
         # cache has { 'key' : (obj, orig_mod_time) }
         self.cache = {}
 


Test patch: