Repository: dask/distributed

Issue:
Use of unix only resource module
Prevents using distributed on Windows!

``` python-traceback
 File "C:/dev/code/sandbox/test_dask.py", line 8, in <module>
    from distributed import Client

  File "c:\dev\packages\distributed\distributed\__init__.py", line 5, in <module>
    from .deploy import LocalCluster

  File "c:\dev\packages\distributed\distributed\deploy\__init__.py", line 3, in <module>
    from .local import LocalCluster

  File "c:\dev\packages\distributed\distributed\deploy\local.py", line 15, in <module>
    from ..scheduler import Scheduler

  File "c:\dev\packages\distributed\distributed\scheduler.py", line 11, in <module>
    import resource

ImportError: No module named 'resource'
```

Issue was introduced in https://github.com/dask/distributed/commit/b2cc90a88be5246231325c773c1d19e7997df3da but is also present on [`master`](https://github.com/dask/distributed/blob/master/distributed/scheduler.py#L11)

It looks like the windows equivalent is `_getmaxstdio`/`_setmaxstdio`?

![image](https://cloud.githubusercontent.com/assets/881019/19098950/04c1a96e-8af5-11e6-88a2-6f1ac814e175.png)
https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx

http://docs.activestate.com/activepython/2.7/pywin32/win32file___getmaxstdio_meth.html
http://docs.activestate.com/activepython/2.7/pywin32/win32file___setmaxstdio_meth.html
![image](https://cloud.githubusercontent.com/assets/881019/19098969/24a61e90-8af5-11e6-8d7d-b42662c99080.png)

![image](https://cloud.githubusercontent.com/assets/881019/19098976/378b2ce4-8af5-11e6-96cf-ad8932bc4dba.png)



Discussion:
It seems to be protected in [`dask_scheduler.py`](https://github.com/dask/distributed/blob/master/distributed/cli/dask_scheduler.py#L55-L58) but not in [`scheduler.py`](https://github.com/dask/distributed/blob/master/distributed/scheduler.py#L321).

Though the top-level [import](https://github.com/dask/distributed/blob/master/distributed/cli/dask_scheduler.py#L8) in `dask_scheduler.py` will barf on Windows so that's broken too.

A short term solution would be to just disable this entirely for Windows
and trust to the 512 default.  This is a bad idea if people are deploying
large clusters on Windows, but a good idea if we want to get things moving
again quickly.

On Wed, Oct 5, 2016 at 6:59 PM, Dave Hirschfeld notifications@github.com
wrote:

> Prevents using distributed on Windows!
> 
>  File "C:/dev/code/sandbox/test_dask.py", line 8, in <module>
>     from distributed import Client
> 
>   File "c:\dev\packages\distributed\distributed__init__.py", line 5, in <module>
>     from .deploy import LocalCluster
> 
>   File "c:\dev\packages\distributed\distributed\deploy__init__.py", line 3, in <module>
>     from .local import LocalCluster
> 
>   File "c:\dev\packages\distributed\distributed\deploy\local.py", line 15, in <module>
>     from ..scheduler import Scheduler
> 
>   File "c:\dev\packages\distributed\distributed\scheduler.py", line 11, in <module>
>     import resource
> ImportError: No module named 'resource'
> 
> Issue was introduced in b2cc90a
> https://github.com/dask/distributed/commit/b2cc90a88be5246231325c773c1d19e7997df3da
> but is also present on master
> https://github.com/dask/distributed/blob/master/distributed/scheduler.py#L11
> 
> It looks like the windows equivalent is _getmaxstdio/_setmaxstdio?
> 
> [image: image]
> https://cloud.githubusercontent.com/assets/881019/19098950/04c1a96e-8af5-11e6-88a2-6f1ac814e175.png
> https://msdn.microsoft.com/en-us/library/kdfaxaay(v=vs.140).aspx
> 
> http://docs.activestate.com/activepython/2.7/pywin32/
> win32file___getmaxstdio_meth.html
> http://docs.activestate.com/activepython/2.7/pywin32/
> win32file___setmaxstdio_meth.html
> [image: image]
> https://cloud.githubusercontent.com/assets/881019/19098969/24a61e90-8af5-11e6-8d7d-b42662c99080.png
> 
> [image: image]
> https://cloud.githubusercontent.com/assets/881019/19098976/378b2ce4-8af5-11e6-96cf-ad8932bc4dba.png
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/dask/distributed/issues/564, or mute the thread
> https://github.com/notifications/unsubscribe-auth/AASszBS7G8q6PPHr9I8AQqupXoL_P74fks5qxCvhgaJpZM4KPYLV
> .

I'd be happy with that (for now) - and in fact that's how I've patched my local version! I guess if anyone ever has a problem with the default of 512 it can be revisited then...

Any interest in submitting a PR with a fix?

On Wed, Oct 5, 2016 at 7:32 PM, Dave Hirschfeld notifications@github.com
wrote:

> I'd be happy with that (for now) - and in fact that's how I've patched my
> local version!
> 
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> https://github.com/dask/distributed/issues/564#issuecomment-251829341,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AASszFfv63mZMAVJU0Mck55ejggh_rVLks5qxDOYgaJpZM4KPYLV
> .

I can give it a go. Might not be until next week though...

@dhirschfeld see this issue: https://github.com/dask/distributed/pull/565. large clusters on windows actually work nicely, provided a linux based scheduler :>


Resolving patch:
diff --git a/distributed/cli/dask_scheduler.py b/distributed/cli/dask_scheduler.py
--- a/distributed/cli/dask_scheduler.py
+++ b/distributed/cli/dask_scheduler.py
@@ -5,7 +5,6 @@
 import logging
 import multiprocessing
 import os
-import resource
 import socket
 import subprocess
 import sys
@@ -53,6 +52,7 @@ def del_pid_file():
         atexit.register(del_pid_file)
 
     if sys.platform.startswith('linux'):
+        import resource   # module fails importing on Windows
         soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
         limit = max(soft, hard // 2)
         resource.setrlimit(resource.RLIMIT_NOFILE, (limit, hard))
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -8,7 +8,6 @@
 import os
 import pickle
 import random
-import resource
 import socket
 from time import time
 from timeit import default_timer
@@ -33,7 +32,8 @@
 from .core import (rpc, connect, read, write, MAX_BUFFER_SIZE,
         Server, send_recv, coerce_to_address, error_message)
 from .utils import (All, ignoring, clear_queue, get_ip, ignore_exceptions,
-        ensure_ip, log_errors, key_split, mean, divide_n_among_bins)
+        ensure_ip, get_fileno_limit, log_errors, key_split, mean,
+        divide_n_among_bins)
 from .config import config
 
 
@@ -318,7 +318,7 @@ def __init__(self, center=None, loop=None,
                  ('released', 'erred'): self.transition_released_erred
         }
 
-        connection_limit = resource.getrlimit(resource.RLIMIT_NOFILE)[0] / 2
+        connection_limit = get_fileno_limit() / 2
 
         super(Scheduler, self).__init__(handlers=self.handlers,
                 max_buffer_size=max_buffer_size, io_loop=self.loop,
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -12,6 +12,11 @@
 import tempfile
 from threading import Thread
 
+try:
+    import resource
+except ImportError:
+    resource = None
+
 from dask import istask
 from toolz import memoize, valmap
 from tornado import gen
@@ -33,6 +38,18 @@ def funcname(func):
         return str(func)
 
 
+def get_fileno_limit():
+    """
+    Get the maximum number of open files per process.
+    """
+    if resource is not None:
+        return resource.getrlimit(resource.RLIMIT_NOFILE)[0]
+    else:
+        # Default ceiling for Windows when using the CRT, though it
+        # is settable using _setmaxstdio().
+        return 512
+
+
 def get_ip(host='8.8.8.8', port=80):
     try:
         return [(s.connect((host, port)), s.getsockname()[0], s.close())


Test patch: