Repository: dask/dask

Issue:
Implement `Array.__iter__`
This came up recently in https://github.com/pandas-dev/pandas/issues/38645 and I took a greedy stab at implementing it in: https://github.com/dask/dask/pull/7888

I abandoned that PR because a bunch of tests failed and I feel like it's at least a little surprising to have `list(array)` return a list of real values. It seems too easy to do by mistake. 

For context, currently on main if you call `list(array)` you get an array of getitem calls with the same length as the 0th dimension of the array. It's possible that is already the best solution. In which case we can probably close this.


Discussion:
What happens if the array has an unknown shape? As an example would consider how `a1` below behaves

```python
import dask.array as da


a0 = da.arange(10, chunks=2)

a1 = a0[a0 % 2 == 0]
```
On main you get 

```python-traceback
ValueError: Arrays chunk sizes are unknown: (nan,)

A possible solution: https://docs.dask.org/en/latest/array-chunks.html#unknown-chunks
Summary: to compute chunks sizes, use

   x.compute_chunk_sizes()  # for Dask Array `x`
   ddf.to_dask_array(lengths=True)  # for Dask DataFrame `ddf`

```
> For context, currently on main if you call `list(array)` you get an array of getitem calls with the same length as the 0th dimension of the array. It's possible that is already the best solution. In which case we can probably close this.

Isn't it the best solution? I read through the linked threads, but I don't see why hiding a dask compute call inside this would be preferred. It seems very non-intuitive for every other circumstance.
Hello, I've been following this with interest, and also agree that the "array of getitem calls with the same length as the 0th" is what I would expect to occur (i.e. _not_ returning computed numpy arrays).

Thanks,
David

Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -1502,6 +1502,10 @@ def name(self, val):
             "please set ``._name``"
         )
 
+    def __iter__(self):
+        for i in range(len(self)):
+            yield self[i]
+
     __array_priority__ = 11  # higher than numpy.ndarray and numpy.matrix
 
     def __array__(self, dtype=None, **kwargs):
diff --git a/dask/dataframe/_compat.py b/dask/dataframe/_compat.py
--- a/dask/dataframe/_compat.py
+++ b/dask/dataframe/_compat.py
@@ -11,6 +11,7 @@
 PANDAS_GT_120 = PANDAS_VERSION >= parse_version("1.2.0")
 PANDAS_GT_121 = PANDAS_VERSION >= parse_version("1.2.1")
 PANDAS_GT_130 = PANDAS_VERSION >= parse_version("1.3.0")
+PANDAS_GT_131 = PANDAS_VERSION >= parse_version("1.3.1")
 
 
 if PANDAS_GT_100:


Test patch:
diff --git a/dask/array/tests/test_array_core.py b/dask/array/tests/test_array_core.py
--- a/dask/array/tests/test_array_core.py
+++ b/dask/array/tests/test_array_core.py
@@ -4210,6 +4210,21 @@ def test_normalize_chunks_nan():
     assert "auto" in str(info.value)
 
 
+def test_pandas_from_dask_array():
+    pd = pytest.importorskip("pandas")
+    from dask.dataframe._compat import PANDAS_GT_130, PANDAS_GT_131
+
+    a = da.ones((12,), chunks=4)
+    s = pd.Series(a, index=range(12))
+
+    if PANDAS_GT_130 and not PANDAS_GT_131:
+        # https://github.com/pandas-dev/pandas/issues/38645
+        assert s.dtype != a.dtype
+    else:
+        assert s.dtype == a.dtype
+        assert_eq(s.values, a)
+
+
 def test_from_zarr_unique_name():
     zarr = pytest.importorskip("zarr")
     a = zarr.array([1, 2, 3])