Repository: dask/dask

Issue:
dask apply loses index type with simple meta
**What happened**:
When trying to *apply* a function to a dataframe, and giving a simple meta, the perceived dtype of the index of the return value is lost. Also the name of the index is lost (harder to show in this minimal example, but pretty similar I assume).
In the bigger picture this gives issues with ddf.merge where it expects the index column "[0]" to exist but instead finds "nameofcolumn".

**What you expected to happen**:
The perceived dtype of the index of the return value is the same as the dtype of the index of return value when computed.

**Minimal Complete Verifiable Example**:

![image](https://user-images.githubusercontent.com/7152733/148384994-daefd5a8-7faf-4a68-a0a5-0960b9a2d0a2.png)

```python
import dask.dataframe as dd
from dask.datasets import timeseries
ddf = timeseries()
ddf.index.dtype                        # shows dtype('<M8[ns]')
editedx = ddf.x.apply(lambda x: x + 1, meta=ddf.x.dtype) # some simple example, giving a simple meta argument
editedx.head()
editedx.index.dtype                  # incorrectly shows dtype('int64'), the default
editedx.index.compute().dtype # correctly shows dtype('<M8[ns]'), as was the dtype of the initial data

```

**Anything else we need to know?**:
It works in this case when we leave the meta out, but in some cases the autodeduction that happens is wrong and we need to provide a specific (if sometimes simple) meta to fix this).

**Environment**:

- Dask version: 2021.12.0
- Python version: 3.9.7
- Operating System: linux (jupyterhub docker on k8s)
- Install method (conda, pip, source): (installed with the base jupyterhub image from above, source I guess? pip freeze shows dask @ file:///home/conda/feedstock_root/build_artifacts/dask-core_1639176667673/work)




Discussion:
Kind-of workaround: provide this as meta:
```python
dask.dataframe.utils.make_meta(ddf).x
```
or, if you need to change a column's type:
```python
dask.dataframe.utils.make_meta(ddf).x.astype("int")
```
or if you need to add a new column:
```python
dask.dataframe.utils.make_meta(ddf).rename({"x":"something"},axis=1)["something"].astype(int)
```
making 
```python
editedx = ddf.x.apply(lambda x: x + 1, meta=ddf.x.dtype)
```
into
```python
editedx = ddf.x.apply(lambda x: x + 1, meta=dask.dataframe.utils.make_meta(ddf).rename({"x":"editedx"},axis=1)["editedx"].astype(int))
```
in the worst case.
Thanks for writing this up @JoranDox and sorry it took a while to get to.

I think the issue here is really around better documentation of meta. The expectation if you pass `meta=` is that that is the only thing that is used to create the meta. The original dataframe is not looked at at all. 

So your workaround is actually the correct way to go about this. Another option is to take a small piece of the data and apply your meta to that:

```python
import dask.dataframe as dd
from dask.datasets import timeseries
ddf = timeseries()

def func(x):
    return x + 1

editedx = ddf.x.apply(func, meta=func(ddf.x.head()))
```

OR set the meta on the original dataframe to something that more closely matches the real data and then the default meta that is created is more likely to be correct:

```python
import dask.dataframe as dd
from dask.datasets import timeseries
ddf = timeseries()
ddf._meta = ddf.head()

editedx = ddf.x.apply(lambda x: x + 1)
```
Wait I might have posted that too soon I am looking more closely at the codebase now.
Yeah I take that whole lecture back. You just need to pass a tuple instead of a plain dtype. That way dask knows to expect a series. If you look at the warning when you don't provide a `meta=` you'll see that is actually what it recommends:

```python
editedx = ddf.x.apply(lambda x: x + 1, meta=("x", ddf.x.dtype))
```
I think that in the case that you originally tried dask should throw an error.

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -6051,8 +6051,10 @@ def map_partitions(
         # Use non-normalized kwargs here, as we want the real values (not
         # delayed values)
         meta = _emulate(func, *args, udf=True, **kwargs)
+        meta_is_emulated = True
     else:
         meta = make_meta(meta, index=meta_index, parent_meta=parent_meta)
+        meta_is_emulated = False
 
     if all(isinstance(arg, Scalar) for arg in args):
         layer = {
@@ -6061,9 +6063,16 @@ def map_partitions(
         graph = HighLevelGraph.from_collections(name, layer, dependencies=args)
         return Scalar(graph, name, meta)
     elif not (has_parallel_type(meta) or is_arraylike(meta) and meta.shape):
-        # If `meta` is not a pandas object, the concatenated results will be a
-        # different type
-        meta = make_meta(_concat([meta]), index=meta_index)
+        if meta_is_emulated:
+            # If `meta` is not a pandas object, the concatenated results will be a
+            # different type
+            meta = make_meta(_concat([meta]), index=meta_index)
+        else:
+            raise ValueError(
+                "Meta is not valid, `map_partitions` expects output to be a pandas object. "
+                "Try passing a pandas object as meta or a dict or tuple representing the "
+                "(name, dtype) of the columns."
+            )
 
     # Ensure meta is empty series
     meta = make_meta(meta, parent_meta=parent_meta)


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -3005,6 +3005,16 @@ def test_apply_warns():
     assert "int64" in str(w[0].message)
 
 
+def test_apply_fails_with_invalid_meta():
+    df = pd.DataFrame({"x": [1, 2, 3, 4], "y": [10, 20, 30, 40]})
+    ddf = dd.from_pandas(df, npartitions=2)
+
+    func = lambda row: row["x"] + row["y"]
+
+    with pytest.raises(ValueError, match="Meta is not valid"):
+        ddf.apply(func, axis=1, meta=int)
+
+
 def test_applymap():
     df = pd.DataFrame({"x": [1, 2, 3, 4], "y": [10, 20, 30, 40]})
     ddf = dd.from_pandas(df, npartitions=2)