Repository: dask/dask

Issue:
Extra progress bar being displayed
### Sample code:
```python
from dask.array import ones
from dask.diagnostics import ProgressBar

data = ones((8,79,200,401),dtype='f4',chunks=(1,1,200,401))
with ProgressBar():
  s = data.sum().compute()

print (s)
```

### Result (version 0.18.1):
```
[########################################] | 100% Completed |  0.4s
[########################################] | 100% Completed |  0.1s
50686400.0
```
### Result (version 0.17.5):
```
[########################################] | 100% Completed |  0.4s
50686400.0
```

<details>
<summary>Version information</summary>

(from <code>xarray.show_versions</code>)

commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.10.0-514.el7.x86_64
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

xarray: 0.10.7
pandas: 0.23.1
numpy: 1.14.5
scipy: 0.13.3
netCDF4: 1.4.0
h5netcdf: None
h5py: 2.2.1
Nio: None
zarr: None
bottleneck: None
cyordereddict: None
dask: 0.18.1
distributed: None
matplotlib: 1.3.1
cartopy: None
seaborn: None
setuptools: 39.2.0
pip: 1.5.4
conda: None
pytest: None
IPython: 1.2.1
sphinx: 1.2.2
</details>


Discussion:
Thank you for the excellent minimal example @neishm .  Hopefully someone comes by soon who has the time to resolve this.  Alternatively, if you're interested in diving in here your efforts would be most welcome.
Traced it to the following change in **dask.base.repack()**:

Version **0.17.5**:
https://github.com/dask/dask/blob/b2bed5eb145b39641181140b8f64a9b4427b53ba/dask/base.py#L279

Version **0.18.1**:
https://github.com/dask/dask/blob/d7755492c81b7b11788a314e008edd11663ff150/dask/base.py#L283

`local.get_sync()` triggers a second progress bar, whereas `simple_get()` does not.  I confirmed this by checking out **0.18.1**, and changing that line of`**dask.base.repack** to:
```python
from .core import get as simple_get
return simple_get(dsk, out)
```
With this I get a single progress bar again.  Don't know what the proper fix is, though.
Hrm, indeed.  Nice find.  Git blame shows that I made this change.  I
suspect that I thought I was just cleaning up.  My guess is that it is safe
to revert this line.  It would also be good to ensure that this doesn't
happen again by having a test that ensures that we only output a single
line.  I think that the capsys pytest fixture can help us with this.

Do you have any interest in submitting a PR with this fix?  The tests to
look at are probably in dask/diagnostics/tests/test_progress.py.

On Mon, Jun 25, 2018 at 1:51 PM, Mike Neish <notifications@github.com>
wrote:

> Traced it to the following change in *dask.base.repack()*:
>
> Version *0.17.5*:
> https://github.com/dask/dask/blob/b2bed5eb145b39641181140b8f64a9
> b4427b53ba/dask/base.py#L279
>
> Version *0.18.1*:
> https://github.com/dask/dask/blob/d7755492c81b7b11788a314e008edd
> 11663ff150/dask/base.py#L283
>
> local.get_sync() triggers a second progress bar, whereas simple_get()
> does not. I confirmed this by checking out *0.18.1*, and changing that
> line of`*dask.base.repack* to:
>
> from .core import get as simple_getreturn simple_get(dsk, out)
>
> With this I get a single progress bar again. Don't know what the proper
> fix is, though.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3668#issuecomment-400038538>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszEqlglSnRAbJ3MqRgXq3n2tjryaoks5uASMigaJpZM4U2c3Y>
> .
>


Resolving patch:
diff --git a/dask/base.py b/dask/base.py
--- a/dask/base.py
+++ b/dask/base.py
@@ -16,7 +16,7 @@
 
 from .compatibility import long, unicode
 from .context import thread_state
-from .core import flatten, quote
+from .core import flatten, quote, get as simple_get
 from .hashing import hash_buffer_hex
 from .utils import Dispatch, ensure_dict
 from . import config, local, threaded
@@ -280,7 +280,7 @@ def _unpack(expr):
     def repack(results):
         dsk = repack_dsk.copy()
         dsk[collections_token] = quote(results)
-        return local.get_sync(dsk, out)
+        return simple_get(dsk, out)
 
     return collections, repack
 


Test patch:
diff --git a/dask/diagnostics/tests/test_progress.py b/dask/diagnostics/tests/test_progress.py
--- a/dask/diagnostics/tests/test_progress.py
+++ b/dask/diagnostics/tests/test_progress.py
@@ -18,11 +18,21 @@
 
 def check_bar_completed(capsys, width=40):
     out, err = capsys.readouterr()
+    assert out.count('100% Completed') == 1
     bar, percent, time = [i.strip() for i in out.split('\r')[-1].split('|')]
     assert bar == '[' + '#' * width + ']'
     assert percent == '100% Completed'
 
 
+def test_array_compute(capsys):
+    from dask.array import ones
+    data = ones((100,100),dtype='f4',chunks=(100,100))
+    with ProgressBar():
+        out = data.sum().compute()
+    assert out == 10000
+    check_bar_completed(capsys)
+
+
 def test_progressbar(capsys):
     with ProgressBar():
         out = get_threaded(dsk, 'e')