Repository: dask/distributed

Issue:
HPC cluster workers preload file not found
I'm launching multiple (in this case 40, but it doesn't matter) DASK workers at the same time on a HPC using a job scheduler. They start running almost at the same time and are preloading the same dask_preload.py file. Some (approximately half of them) workers die immediately with the following error, the others start up correctly:

```
Traceback (most recent call last):
  File "/python_virtualenv/bin/dask-worker", line 11, in <module>
    sys.exit(go())
  File "/python_virtualenv/lib/python2.7/site-packages/distributed/cli/dask_worker.py", line 252, in go
    main()
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 696, in main
    with self.make_context(prog_name, args, **extra) as ctx:
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 621, in make_context
    self.parse_args(ctx, args)
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 880, in parse_args
    value, args = param.handle_parse_result(ctx, opts, args)
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 1404, in handle_parse_result
    self.callback, ctx, self, value)
  File "/python_virtualenv/lib/python2.7/site-packages/click/core.py", line 78, in invoke_param_callback
    return callback(ctx, param, value)
  File "/python_virtualenv/lib/python2.7/site-packages/distributed/preloading.py", line 32, in validate_preload_argv
    preload_modules = _import_modules(ctx.params.get("preload"))
  File "/python_virtualenv/lib/python2.7/site-packages/distributed/preloading.py", line 92, in _import_modules
    module = import_file(name)[0]
  File "/python_virtualenv/lib/python2.7/site-packages/distributed/utils.py", line 1003, in import_file
    os.remove(cache_file)
OSError: [Errno 2] No such file or directory: 'dask_preload.pyc'
```

I can schedule workers to start 1 at a time. But first I want to know if it is possible to fix this within the scheduler code itself?

Btw. If I start 1 worker, it is able to load the dask_preload.py file.


Discussion:
Interesting!  My first guess would be some sort of race condition with the file system and the Python import system?  I would be curious to know if this issue persists in Python 3.
Python 3 gives me an error alike the one of Python 2. 
`FileNotFoundError: [Errno 2] No such file or directory: 'dask_preload.cpython-35.pyc'`
OK, not fixed in Python 2 then.  Again, my guess is something related to
slow/inconsistent network file system and race conditions with Python
imports.

@pitrou might find this problem interesting (though he may not have the
time to actually solve it)

On Mon, Apr 16, 2018 at 3:00 AM, Wouter van Roosmalen <
notifications@github.com> wrote:

> Python 3 gives me an error alike the one of Python 2.
> FileNotFoundError: [Errno 2] No such file or directory:
> '/hpc/data/ada4euv/users/wroosmal/csi_ada/ada_lib/doc/
> hpc/__pycache__/dask_preload.cpython-35.pyc'
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/1915#issuecomment-381498962>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszEKdn_YRYJFGAalhSBZMcSGobPskks5tpEGOgaJpZM4TRkBV>
> .
>

This looks like a benign case of race condition in https://github.com/dask/distributed/blob/master/distributed/utils.py#L1000-L1003.

Just replace:
```python
if os.path.exists(cache_file):
    os.remove(cache_file)
```
with:
```python
try:
    os.remove(cache_file)
except OSError:
    pass  # ignore "file not found"
```
(you can refine the exception check by querying errno)
Ah, I'm glad to see this.  I'm curious though, what happens if someone gives us a pyc file directly?  Should we only be removing this file if they ask us to import a `.py` file?
What would be the case for giving us a pyc file? I think the only reason for that would be proprietary software where the source code isn't supplied, in which case you probably don't want to remove the pyc file :-)
Great.  Just checking :)  Thanks for the quick answer @pitrou !
Is this something to change in a next release? Or a custom change specific to this use case?
Yes, what @pitrou proposes is an easy fix.  I have a fix locally that I'll
push up in a bit.

On Tue, Apr 17, 2018 at 2:08 AM, Wouter van Roosmalen <
notifications@github.com> wrote:

> Is this something to change in a next release? Or a custom change specific
> to this use case?
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/1915#issuecomment-381857309>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszBv1_dc_eQuI2YmpXk1QmO9aa9Ifks5tpYbwgaJpZM4TRkBV>
> .
>


Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -2480,13 +2480,13 @@ def upload_file(self, filename, **kwargs):
         """ Upload local package to workers
 
         This sends a local file up to all worker nodes.  This file is placed
-        into a temporary directory on Python's system path so any .py, .pyc, .egg
+        into a temporary directory on Python's system path so any .py,  .egg
         or .zip  files will be importable.
 
         Parameters
         ----------
         filename: string
-            Filename of .py, .pyc, .egg or .zip file to send to workers
+            Filename of .py, .egg or .zip file to send to workers
 
         Examples
         --------
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -987,19 +987,19 @@ def open_port(host=''):
 
 
 def import_file(path):
-    """ Loads modules for a file (.py, .pyc, .zip, .egg) """
+    """ Loads modules for a file (.py, .zip, .egg) """
     directory, filename = os.path.split(path)
     name, ext = os.path.splitext(filename)
     names_to_import = []
     tmp_python_path = None
 
-    if ext in ('.py', '.pyc'):
+    if ext in ('.py'):  # , '.pyc'):
         if directory not in sys.path:
             tmp_python_path = directory
         names_to_import.append(name)
-        # Ensures that no pyc file will be reused
+    if ext == '.py':  # Ensure that no pyc file will be reused
         cache_file = cache_from_source(path)
-        if os.path.exists(cache_file):
+        with ignoring(OSError):
             os.remove(cache_file)
     if ext in ('.egg', '.zip'):
         if path not in sys.path:


Test patch:
diff --git a/distributed/tests/test_worker.py b/distributed/tests/test_worker.py
--- a/distributed/tests/test_worker.py
+++ b/distributed/tests/test_worker.py
@@ -20,7 +20,7 @@
 
 from distributed import (Nanny, Client, get_client, wait, default_client,
         get_worker, Reschedule)
-from distributed.compatibility import WINDOWS
+from distributed.compatibility import WINDOWS, cache_from_source
 from distributed.config import config
 from distributed.core import rpc
 from distributed.client import wait
@@ -181,6 +181,33 @@ def g():
     assert not os.path.exists(os.path.join(a.local_dir, 'foobar.py'))
 
 
+@pytest.mark.xfail(reason="don't yet support uploading pyc files")
+@gen_cluster(client=True, ncores=[('127.0.0.1', 1)])
+def test_upload_file_pyc(c, s, w):
+    with tmpfile() as dirname:
+        os.mkdir(dirname)
+        with open(os.path.join(dirname, 'foo.py'), mode='w') as f:
+            f.write('def f():\n    return 123')
+
+        sys.path.append(dirname)
+        try:
+            import foo
+            assert foo.f() == 123
+            pyc = cache_from_source(os.path.join(dirname, 'foo.py'))
+            assert os.path.exists(pyc)
+            yield c.upload_file(pyc)
+
+            def g():
+                import foo
+                return foo.x
+
+            future = c.submit(g)
+            result = yield future
+            assert result == 123
+        finally:
+            sys.path.remove(dirname)
+
+
 @gen_cluster(client=True)
 def test_upload_egg(c, s, a, b):
     eggname = 'mytestegg-1.0.0-py3.4.egg'