Repository: dask/dask

Issue:
Add optional IO-subgraph to Blockwise Layers
**Big Picture**

`Blockwise`  HighLevelGraph (HLG) layers currently *require* a collection dependency.  This has prevented IO (data-generation) operations from being represeted as `Blockwise` objects.  This, in turn, has prevented `optimize_blockwise` from fusing IO tasks with follow-up block-wise transformation.  This is fine if/when we can use `fuse` on the full task-graph dictionary, but not if we hope to [send HLGs directly to the scheduler](https://github.com/dask/distributed/pull/4140).

This PR makes it possible to perform IO from within a `Blockwise` layer (by passing in a special IO subgraph), and introduces a new `BlockwiseParquet` Layer.  These changes demonstrate that `optimize_blockwise` behave correctly for IO-enabled layers, without complicating `optimize_read_parquet_getitem`.

**TODO**:
- [x] Add low-level fusion of blockwise io and transition tasks
- [x] Fuse IO into `subgraph_callable` (*may* make future optimizations cleaner)
- [ ] ~Try to avoid IO-subgraph dict materialization when culling and/or getting dependencies~
- [x] Try to avoid "no-op"  tasks in IO-only Blockwise layers
- [x] Update other IO operations to leverage Blockwise (e.g. `read_csv`)
- [x] Remove `from_callable` API, and remove or update the related test



Discussion:

Resolving patch:
diff --git a/dask/dataframe/multi.py b/dask/dataframe/multi.py
--- a/dask/dataframe/multi.py
+++ b/dask/dataframe/multi.py
@@ -984,7 +984,13 @@ def stack_partitions(dfs, divisions, join="outer"):
     return new_dd_object(dsk, name, meta, divisions)
 
 
-def concat(dfs, axis=0, join="outer", interleave_partitions=False):
+def concat(
+    dfs,
+    axis=0,
+    join="outer",
+    interleave_partitions=False,
+    ignore_unknown_divisions=False,
+):
     """ Concatenate DataFrames along rows.
 
     - When axis=0 (default), concatenate DataFrames row-wise:
@@ -1013,6 +1019,9 @@ def concat(dfs, axis=0, join="outer", interleave_partitions=False):
     interleave_partitions : bool, default False
         Whether to concatenate DataFrames ignoring its order. If True, every
         divisions are concatenated each by each.
+    ignore_unknown_divisions : bool, default False
+        By default a warning is raised if any input has unknown divisions.
+        Set to True to disable this warning.
 
     Notes
     -----
@@ -1057,6 +1066,12 @@ def concat(dfs, axis=0, join="outer", interleave_partitions=False):
     >>> dd.concat([a, b])                               # doctest: +SKIP
     dd.DataFrame<concat-..., divisions=(None, None, None, None)>
 
+    By default concatenating with unknown divisions will raise a warning.
+    Set ``ignore_unknown_divisions=True`` to disable this:
+
+    >>> dd.concat([a, b], ignore_unknown_divisions=True)# doctest: +SKIP
+    dd.DataFrame<concat-..., divisions=(None, None, None, None)>
+
     Different categoricals are unioned
 
     >> dd.concat([                                     # doctest: +SKIP
@@ -1090,12 +1105,13 @@ def concat(dfs, axis=0, join="outer", interleave_partitions=False):
             and all(not df.known_divisions for df in dfs)
             and len({df.npartitions for df in dasks}) == 1
         ):
-            warnings.warn(
-                "Concatenating dataframes with unknown divisions.\n"
-                "We're assuming that the indexes of each dataframes"
-                " are \n aligned. This assumption is not generally "
-                "safe."
-            )
+            if not ignore_unknown_divisions:
+                warnings.warn(
+                    "Concatenating dataframes with unknown divisions.\n"
+                    "We're assuming that the indexes of each dataframes"
+                    " are \n aligned. This assumption is not generally "
+                    "safe."
+                )
             return concat_unindexed_dataframes(dfs)
         else:
             raise ValueError(


Test patch:
diff --git a/dask/dataframe/tests/test_multi.py b/dask/dataframe/tests/test_multi.py
--- a/dask/dataframe/tests/test_multi.py
+++ b/dask/dataframe/tests/test_multi.py
@@ -1406,6 +1406,10 @@ def test_concat_unknown_divisions():
     with pytest.raises(ValueError):
         dd.concat([aa, cc], axis=1)
 
+    with pytest.warns(None) as record:
+        dd.concat([aa, bb], axis=1, ignore_unknown_divisions=True)
+        assert len(record) == 0
+
 
 def test_concat_unknown_divisions_errors():
     a = pd.Series([1, 2, 3, 4, 5, 6])