Repository: dask/dask

Issue:
[FEA] Add official mechanism to check if query-planning is enabled in ``dask.dataframe``
The value of the `"dataframe.query-planning"` config currently dictates whether or not `dask.dataframe` will use dask-expr under the hood. However, after `dask.dataframe` is first imported, the value of the `"dataframe.query-planning"` config becomes somewhat useless.

For example, a down-stream library may wish to warn/raise if `dask.dataframe` is already imported with query-planning enabled, but the value of the `"dataframe.query-planning"` config will typically be `None` whether the module was already imported or not.

One way to mitigate this problem is for `dask.dataframe.__init__` to always convert `None` values into `True` (using the config system). This way, a down-stream library (or user) can interpret `None` as a confirmation that the query-planning config can still be modified. An alternative solution is to tell users to inspect `sys.modules` themselves to confirm that `dask.dataframe` is missing.

Whether or not we introduce a simple way for users to check if the query-planning config is already established (i.e. whether `dask.dataframe` was imported), it still seems problematic that the user **can** change the `"query-planning"` config after the fact. This problem is certainly tricky to resolve, but it would be nice if the user could be warned that their config will not be accommodated.


Discussion:
This kind of problem occurs every now and then for various kinds of configs. Typically we are setting some global variable in this case.
>This kind of problem occurs every now and then for various kinds of configs. Typically we are setting some global variable in this case.

Thanks Florian! Just to clarify: This means you might be in support of having this issue addressed with a global variable that indicates whether or not query-planning is enabled?
Yes, I'm fine with a constant

cc @phofl 

Resolving patch:
diff --git a/dask/dataframe/__init__.py b/dask/dataframe/__init__.py
--- a/dask/dataframe/__init__.py
+++ b/dask/dataframe/__init__.py
@@ -5,19 +5,37 @@
 
 from packaging.version import Version
 
+# The "dataframe.query-planning" config can only be processed once
+DASK_EXPR_ENABLED: bool | None = None
+
 
 def _dask_expr_enabled() -> bool:
     import pandas as pd
 
     import dask
 
+    global DASK_EXPR_ENABLED
+
     use_dask_expr = dask.config.get("dataframe.query-planning")
+    if DASK_EXPR_ENABLED is not None:
+        if (use_dask_expr is True and DASK_EXPR_ENABLED is False) or (
+            use_dask_expr is False and DASK_EXPR_ENABLED is True
+        ):
+            warnings.warn(
+                "The 'dataframe.query-planning' config is now set to "
+                f"{use_dask_expr}, but query planning is already "
+                f"{'enabled' if DASK_EXPR_ENABLED else 'disabled'}. "
+                "The query-planning config can only be changed before "
+                "`dask.dataframe` is first imported!"
+            )
+        return DASK_EXPR_ENABLED
+
     if (
         use_dask_expr is False
         or use_dask_expr is None
         and Version(pd.__version__).major < 2
     ):
-        return False
+        return (DASK_EXPR_ENABLED := False)
     try:
         import dask_expr  # noqa: F401
     except ImportError:
@@ -29,10 +47,10 @@ def _dask_expr_enabled() -> bool:
 """
         if use_dask_expr is None:
             warnings.warn(msg, FutureWarning)
-            return False
+            return (DASK_EXPR_ENABLED := False)
         else:
             raise ImportError(msg)
-    return True
+    return (DASK_EXPR_ENABLED := True)
 
 
 try:


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -6476,3 +6476,13 @@ def test_enforce_runtime_divisions():
             RuntimeError, match="`enforce_runtime_divisions` failed for partition 1"
         ):
             ddf.enforce_runtime_divisions().compute()
+
+
+def test_query_planning_config_warns():
+    # Make sure dd._dask_expr_enabled() warns if the current
+    # "dataframe.query-planning" config conflicts with the
+    # global dd.DASK_EXPR_ENABLED setting.
+    with dask.config.set({"dataframe.query-planning": not DASK_EXPR_ENABLED}):
+        expect = "enabled" if dd.DASK_EXPR_ENABLED else "disabled"
+        with pytest.warns(match=f"query planning is already {expect}"):
+            dd._dask_expr_enabled()