Repository: dask/distributed

Issue:
Dask distributed doc review
# Setup Network
## Using the Python API

"Alternatively, replace Worker with Nanny if you want your workers to be
managed in a separate process by a local nanny process.": perhaps this
sentence could explain why one would want to do such a thing.  Because
of the GIL?
## Software Environment

Perhaps it would be worth mentioning conda environment files
(`conda export -e` or `conda export --explicit`, I'm not sure
which one) and / or conda "kapsels"?
# Client
## Dask

"For more information see the page on Managing Computation":
it seems the latter words should be a hyperlink pointing to the
corresponding doc page.
# API
## Client.publish_dataset

The examples are misformatted.
## Client.replicate

Is this a user-facing API?  The documentation describes technicalities
of the function but does not explain its purpose.  Is it useful for
persisting in-memory data in the face of flaky workers?
## Client.scatter

Similar question as for Client.replicate.  Also, the last examples are
misformatted.
## Client.shutdown

Does this method wait for all tasks to finish or do they get cancelled?
# Data Locality
## Specify workers with Compute/Persist

"Set provide a list of collection to allow_other_workers=[...]
to say that the keys for only some of the collections are loose.":
the beginning of the sentence looks incorrect.
# Managing Memory
## Creating Futures

"Submit and map handle raw Python functions.": some formatting around
method names is missing (should be "`submit` and `map`").
## Resilience

This paragraph does not address scattered data, and could perhaps be
suppressed in favour of the immediately following documentation page
"Resilience".
# Resilience
## Closed Network Connections

"Data scattered out to the workers is not kept in the scheduler [...]":
there is some formatting bug here (some misplaced "``" markup perhaps).
# Scheduling Policies
## Choosing Workers

"Otherwise, if a worker has user-provided restrictions (must run on a
machine with a GPU) [...]":

probably should be "if a task has user-provided restrictions"?
## Last in, first out

"This often conflicts with the first-come-first-served objective but
often results in longer total runtimes and significantly reduced memory
footprints."

Should that be "shorter total runtimes"?
# Scheduling State
## stacks

Does the name mean tasks with affinities are consumed last-in first-out?
## rprocessing

I'm a bit confused here.  Does it mean a given task can run on several
workers at once?  Until here I was assuming a task is an atomic unit of
work (such as a function and its arguments).
## suspicious_tasks

"If a worker is involved in three failures (or some other fixed constant)
then we mark the task as failed"

Should that be "If a task is involved in three failures"?
## Task `key` finished by `worker`

Correct me if I'm wrong, but the snippet seems to lack some code to move
some tasks from the waiting state to the ready state.  I suppose when
iterating over `dependents[key]`, if `waiting[dep]` becomes empty
then `dep` should be switched to the ready state (queued either in
`ready` or `stacks`).  Also, to uphold the last-in first-out property,
it has be done before choosing the next task to run on the worker.
# Worker
## API Documentation

The example does not seem to work (also the `yield` line at the
interpreter prompt looks suspicious).  I get::

```
>>> from distributed import Scheduler, Worker
>>> c = Scheduler('192.168.0.100', 8786)
>>> w = Worker(c.ip, c.port)
Traceback (most recent call last):
  File "<ipython-input-3-ba8fc125e8f6>", line 1, in <module>
    w = Worker(c.ip, c.port)
  File "/home/antoine/distributed/distributed/core.py", line 148, in port
    raise OSError("Server has no port.  Please call .listen first")
OSError: Server has no port.  Please call .listen first
```



Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -538,7 +538,18 @@ def _shutdown(self, fast=False):
                 self.scheduler.close_rpc()
 
     def shutdown(self, timeout=10):
-        """ Send shutdown signal and wait until scheduler terminates """
+        """ Send shutdown signal and wait until scheduler terminates
+
+        This cancels all currently running tasks, clears the state of the
+        scheduler, and shuts down all workers and scheduler.
+
+        You do not need to call this when you finish your session.  You only
+        need to call this if you want to take down the distributed cluster.
+
+        See Also
+        --------
+        Client.restart
+        """
         if self.status == 'closed':
             return
         self.status = 'closed'
@@ -938,6 +949,11 @@ def _threaded_scatter(self, q_or_i, qout, **kwargs):
     def scatter(self, data, workers=None, broadcast=False, maxsize=0):
         """ Scatter data into distributed memory
 
+        This moves data from the local client process into the workers of the
+        distributed scheduler.  Note that it is often better to submit jobs to
+        your workers to have them load the data rather than loading data
+        locally and then scattering it out to them.
+
         Parameters
         ----------
         data: list, iterator, dict, or Queue
@@ -969,14 +985,17 @@ def scatter(self, data, workers=None, broadcast=False, maxsize=0):
          'z': <Future: status: finished, key: z>}
 
         Constrain location of data to subset of workers
+
         >>> c.scatter([1, 2, 3], workers=[('hostname', 8788)])   # doctest: +SKIP
 
         Handle streaming sequences of data with iterators or queues
+
         >>> seq = c.scatter(iter([1, 2, 3]))  # doctest: +SKIP
         >>> next(seq)  # doctest: +SKIP
         <Future: status: finished, key: c0a8a20f903a4915b94db8de3ea63195>,
 
         Broadcast data to all workers
+
         >>> [future] = c.scatter([element], broadcast=True)  # doctest: +SKIP
 
         See Also
@@ -1051,11 +1070,13 @@ def publish_dataset(self, **kwargs):
         Examples
         --------
         Publishing client:
+
         >>> df = dd.read_csv('s3://...')  # doctest: +SKIP
         >>> df = c.persist(df) # doctest: +SKIP
         >>> c.publish_dataset(my_dataset=df)  # doctest: +SKIP
 
         Receiving client:
+
         >>> c.list_datasets()  # doctest: +SKIP
         ['my_dataset']
         >>> df2 = c.get_dataset('my_dataset')  # doctest: +SKIP
@@ -1599,11 +1620,12 @@ def _replicate(self, futures, n=None, workers=None, branching_factor=2):
     def replicate(self, futures, n=None, workers=None, branching_factor=2):
         """ Set replication of futures within network
 
-        This performs a tree copy of the data throughout the network
-        individually on each piece of data.
+        Copy data onto many workers.  This helps to broadcast frequently
+        accessed data and it helps to improve resilience.
 
-        This operation blocks until complete.  It does not guarantee
-        replication of data to future workers.
+        This performs a tree copy of the data throughout the network
+        individually on each piece of data.  This operation blocks until
+        complete.  It does not guarantee replication of data to future workers.
 
         Parameters
         ----------
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -85,25 +85,19 @@ class Worker(Server):
     Examples
     --------
 
-    Create schedulers and workers in Python:
-
-    >>> from distributed import Scheduler, Worker
-    >>> c = Scheduler('192.168.0.100', 8786)  # doctest: +SKIP
-    >>> w = Worker(c.ip, c.port)  # doctest: +SKIP
-    >>> yield w._start(port=8786)  # doctest: +SKIP
-
-    Or use the command line::
+    Use the command line to start a worker::
 
         $ dask-scheduler
         Start scheduler at 127.0.0.1:8786
 
         $ dask-worker 127.0.0.1:8786
-        Start worker at:               127.0.0.1:8786
-        Registered with scheduler at:  127.0.0.1:8787
+        Start worker at:               127.0.0.1:1234
+        Registered with scheduler at:  127.0.0.1:8786
 
     See Also
     --------
-    distributed.scheduler.Scheduler:
+    distributed.scheduler.Scheduler
+    distributed.nanny.Nanny
     """
 
     def __init__(self, scheduler_ip, scheduler_port, ip=None, ncores=None,


Test patch: