Repository: dask/dask

Issue:
Computed array dtype is different from collection dtype when using ufuncs (blockwise)
**What happened**:

This surfaced while investigating #7651, when setting `dtype=` in a ufunc, the collection has the expected dtype, but after computation the dtype may be different. This also surfaces when using `astype`

**What you expected to happen**:

After calling compute the dtype of the resulting `np.ndarray` is the same as the `da.Array`

**Minimal Complete Verifiable Example**:

```python
In [20]: x, y = da.arange(8), da.arange(8)

In [21]: x.dtype, y.dtype
Out[21]: (dtype('int64'), dtype('int64'))

In [22]: da.add(x, y, dtype='f8')
Out[22]: dask.array<add, shape=(8,), dtype=float64, chunksize=(8,), chunktype=numpy.ndarray>

In [23]: z = da.add(x, y, dtype='f8')

In [24]: z.dtype
Out[24]: dtype('float64')
```
at this point everything is as expected, but after computing the dtype is not conserved:

```python
In [25]: z.compute().dtype
Out[25]: dtype('int64')
```

**Anything else we need to know?**:

I'm starting to investigate by looking in blockwise, any input welcome!

**Environment**:

- Dask version: 2021.6.0+7.g738f53fa
- Python version: 3.9
- Operating System: macOS
- Install method (conda, pip, source): pip



Discussion:
My first question is what is the intent of `dtype`? I assume that it is to coerce the result to that type. That is what the numpy docs say: 

> dtype
>
> New in version 1.6.
> 
> Overrides the dtype of the calculation and output arrays. Similar to signature.

[ref](https://numpy.org/doc/stable/reference/ufuncs.html#ufuncs-kwargs)

So that's how `dtype` is different from `meta`. `dtype` is how you say, "and I want the output to be a float" and `meta` is how you tell Dask what to expect from the function. This is probably obvious, but it's helping me to write it out :)

So I messed around a bit and I think you can just set:

```python
need_enforce_dtype = True
``` 
here: https://github.com/dask/dask/blob/35c926c9c4a1df3668b77945351d977261810012/dask/array/core.py#L4306-L4307


Resolving patch:
diff --git a/dask/array/core.py b/dask/array/core.py
--- a/dask/array/core.py
+++ b/dask/array/core.py
@@ -4311,6 +4311,7 @@ def elemwise(op, *args, **kwargs):
 
     need_enforce_dtype = False
     if "dtype" in kwargs:
+        need_enforce_dtype = True
         dt = kwargs["dtype"]
     else:
         # We follow NumPy's rules for dtype promotion, which special cases


Test patch:
diff --git a/dask/array/tests/test_ufunc.py b/dask/array/tests/test_ufunc.py
--- a/dask/array/tests/test_ufunc.py
+++ b/dask/array/tests/test_ufunc.py
@@ -474,3 +474,19 @@ def test_divmod():
     expected = divmod(arr1, arr2)
     assert_eq(result[0], expected[0])
     assert_eq(result[1], expected[1])
+
+
+@pytest.mark.parametrize("dt", ["float64", "float32", "int32", "int64"])
+def test_dtype_kwarg(dt):
+    arr1 = np.array([1, 2, 3])
+    arr2 = np.array([4, 5, 6])
+
+    darr1 = da.from_array(arr1)
+    darr2 = da.from_array(arr2)
+
+    expected = np.add(arr1, arr2, dtype=dt)
+    result = np.add(darr1, darr2, dtype=dt)
+    assert_eq(expected, result)
+
+    result = da.add(darr1, darr2, dtype=dt)
+    assert_eq(expected, result)