Repository: apache/datafusion
Pull request #15594: Set DataFusion runtime configurations through SQL interface
Author: login: kumarlokesh
Labels: name: documentation
description: Improvements or additions to documentation
name: enhancement
description: New feature or request
name: development-process
description: Related to development process of DataFusion
name: core
description: Core DataFusion crate
name: execution
description: Related to the execution crate
Created: 2025-04-05T15:28:52Z | Merged: 2025-04-17T03:30:13Z

## Which issue does this PR close?

<!--
We generally require a GitHub issue to be filed for all bug fixes and enhancements and this helps us generate change logs for our releases. You can link an issue to this PR using the GitHub syntax. For example `Closes #123` indicates that this PR will close issue #123.
-->

- Closes #15552.

## Rationale for this change

<!--
 Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
 Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.  
-->

## What changes are included in this PR?

<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
1. Extended the session context to handle runtime configurations.
2. Currently support memory limit configuration by using `SET datafusion.runtime.memory_limit = '100M'`.
3. Updated documentation.

## Are these changes tested?

<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code

If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->

## Are there any user-facing changes?

<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->

<!--
If there are any breaking changes to public APIs, please add the `api change` label.
-->


Linked issue #123: Implement physical plan for EXISTS subquery
*Note*: migrated from original JIRA: https://issues.apache.org/jira/browse/ARROW-10819

The TPC-H queries include use of the EXISTS which is used to test for the existence of any record in a subquery. For example:

```sql
and *exists* (
    select
        *
    from
        lineitem
    where
        l_orderkey = o_orderkey
        and l_commitdate < l_receiptdate
)
```
Linked issue comments: user: login: alamb
created_at: 2021-04-26T13:23:43Z
body: Comment from Andy Grove(andygrove) @ 2020-12-31T19:42:45.132+0000:
<pre>The example given here is a correlated subquery that can be translated into a join.

Here is a random stackoverflow discussion on this for reference (I have not reviewed it)

https://stackoverflow.com/questions/1772609/procedurally-transform-subquery-into-join</pre>
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2022-10-21T19:21:30Z
body: In case anyone is curious -- we support correlated versions of these queries (via a join) but if there is no correlation (not super useful) we do not

```sql
❯ create table foo as select * from (values (1), (2), (NULL)) as sql
;
0 rows in set. Query took 0.022 seconds.
3 rows in set. Query took 0.007 seconds.
❯ create table bar as select * from (values (1), (2), (NULL)) as sql;
0 rows in set. Query took 0.000 seconds.
❯ select * from foo where exists (select column1 from bar);
NotImplemented("Physical plan does not support logical expression EXISTS (<subquery>)")
❯ select * from foo where exists (select column1 from bar where foo.column1 = bar.column1);
+---------+
| column1 |
+---------+
| 2       |
| 1       |
+---------+
```
author_association: CONTRIBUTOR
user: login: logan-keede
created_at: 2025-02-09T19:53:56Z
body: 
> In case anyone is curious -- we support correlated versions of these queries (via a join) but if there is no correlation (not super useful) we do not
> 
> ❯ create table foo as select * from (values (1), (2), (NULL)) as sql
> ;
> 0 rows in set. Query took 0.022 seconds.
> 3 rows in set. Query took 0.007 seconds.
> ❯ create table bar as select * from (values (1), (2), (NULL)) as sql;
> 0 rows in set. Query took 0.000 seconds.
> ❯ select * from foo where exists (select column1 from bar);
> NotImplemented("Physical plan does not support logical expression EXISTS (<subquery>)")
> ❯ select * from foo where exists (select column1 from bar where foo.column1 = bar.column1);
> +---------+
> | column1 |
> +---------+
> | 2       |
> | 1       |
> +---------+

```sql

> explain select * from foo where exists (select column1 from bar);
+---------------+-----------------------------------------------------+
| plan_type     | plan                                                |
+---------------+-----------------------------------------------------+
| logical_plan  | LeftSemi Join:                                      |
|               |   TableScan: foo projection=[column1]               |
|               |   SubqueryAlias: __correlated_sq_1                  |
|               |     TableScan: bar projection=[]                    |
| physical_plan | NestedLoopJoinExec: join_type=RightSemi             |
|               |   DataSourceExec: partitions=1, partition_sizes=[1] |
|               |   DataSourceExec: partitions=1, partition_sizes=[1] |
|               |                                                     |
+---------------+-----------------------------------------------------+
2 row(s) fetched. 
Elapsed 0.007 seconds.

> select * from foo where exists (select column1 from bar);
+---------+
| column1 |
+---------+
| 1       |
| 2       |
| NULL    |
+---------+
3 row(s) fetched. 
Elapsed 0.006 seconds.
```


I get this on `datafusion 45.0.0`, perhaps this issue can be closed or at least updated if there is a case where we still don't support EXISTS in physical plan.

author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-02-10T15:04:31Z
body: Looks good -- thanks for checking @logan-keede  -- we can open a new issue if we find another hole
author_association: CONTRIBUTOR

Diff:
diff --git a/.github/workflows/rust.yml b/.github/workflows/rust.yml
index 3fa8ce0804749..f3b7e19a4970b 100644
--- a/.github/workflows/rust.yml
+++ b/.github/workflows/rust.yml
@@ -693,6 +693,11 @@ jobs:
           # If you encounter an error, run './dev/update_function_docs.sh' and commit
           ./dev/update_function_docs.sh
           git diff --exit-code
+      - name: Check if runtime_configs.md has been modified
+        run: |
+          # If you encounter an error, run './dev/update_runtime_config_docs.sh' and commit
+          ./dev/update_runtime_config_docs.sh
+          git diff --exit-code
 
   # Verify MSRV for the crates which are directly used by other projects:
   # - datafusion
diff --git a/datafusion/core/src/bin/print_runtime_config_docs.rs b/datafusion/core/src/bin/print_runtime_config_docs.rs
new file mode 100644
index 0000000000000..f374a5acb78a0
--- /dev/null
+++ b/datafusion/core/src/bin/print_runtime_config_docs.rs
@@ -0,0 +1,23 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+use datafusion_execution::runtime_env::RuntimeEnvBuilder;
+
+fn main() {
+    let docs = RuntimeEnvBuilder::generate_config_markdown();
+    println!("{}", docs);
+}
diff --git a/datafusion/core/src/execution/context/mod.rs b/datafusion/core/src/execution/context/mod.rs
index fc110a0699df2..0bb91536da3ca 100644
--- a/datafusion/core/src/execution/context/mod.rs
+++ b/datafusion/core/src/execution/context/mod.rs
@@ -35,7 +35,11 @@ use crate::{
     },
     datasource::{provider_as_source, MemTable, ViewTable},
     error::{DataFusionError, Result},
-    execution::{options::ArrowReadOptions, runtime_env::RuntimeEnv, FunctionRegistry},
+    execution::{
+        options::ArrowReadOptions,
+        runtime_env::{RuntimeEnv, RuntimeEnvBuilder},
+        FunctionRegistry,
+    },
     logical_expr::AggregateUDF,
     logical_expr::ScalarUDF,
     logical_expr::{
@@ -1036,13 +1040,73 @@ impl SessionContext {
             variable, value, ..
         } = stmt;
 
-        let mut state = self.state.write();
-        state.config_mut().options_mut().set(&variable, &value)?;
-        drop(state);
+        // Check if this is a runtime configuration
+        if variable.starts_with("datafusion.runtime.") {
+            self.set_runtime_variable(&variable, &value)?;
+        } else {
+            let mut state = self.state.write();
+            state.config_mut().options_mut().set(&variable, &value)?;
+            drop(state);
+        }
 
         self.return_empty_dataframe()
     }
 
+    fn set_runtime_variable(&self, variable: &str, value: &str) -> Result<()> {
+        let key = variable.strip_prefix("datafusion.runtime.").unwrap();
+
+        match key {
+            "memory_limit" => {
+                let memory_limit = Self::parse_memory_limit(value)?;
+
+                let mut state = self.state.write();
+                let mut builder =
+                    RuntimeEnvBuilder::from_runtime_env(state.runtime_env());
+                builder = builder.with_memory_limit(memory_limit, 1.0);
+                *state = SessionStateBuilder::from(state.clone())
+                    .with_runtime_env(Arc::new(builder.build()?))
+                    .build();
+            }
+            _ => {
+                return Err(DataFusionError::Plan(format!(
+                    "Unknown runtime configuration: {}",
+                    variable
+                )))
+            }
+        }
+        Ok(())
+    }
+
+    /// Parse memory limit from string to number of bytes
+    /// Supports formats like '1.5G', '100M', '512K'
+    ///
+    /// # Examples
+    /// ```
+    /// use datafusion::execution::context::SessionContext;
+    ///
+    /// assert_eq!(SessionContext::parse_memory_limit("1M").unwrap(), 1024 * 1024);
+    /// assert_eq!(SessionContext::parse_memory_limit("1.5G").unwrap(), (1.5 * 1024.0 * 1024.0 * 1024.0) as usize);
+    /// ```
+    pub fn parse_memory_limit(limit: &str) -> Result<usize> {
+        let (number, unit) = limit.split_at(limit.len() - 1);
+        let number: f64 = number.parse().map_err(|_| {
+            DataFusionError::Plan(format!(
+                "Failed to parse number from memory limit '{}'",
+                limit
+            ))
+        })?;
+
+        match unit {
+            "K" => Ok((number * 1024.0) as usize),
+            "M" => Ok((number * 1024.0 * 1024.0) as usize),
+            "G" => Ok((number * 1024.0 * 1024.0 * 1024.0) as usize),
+            _ => Err(DataFusionError::Plan(format!(
+                "Unsupported unit '{}' in memory limit '{}'",
+                unit, limit
+            ))),
+        }
+    }
+
     async fn create_custom_table(
         &self,
         cmd: &CreateExternalTable,
@@ -1833,7 +1897,6 @@ mod tests {
     use crate::test;
     use crate::test_util::{plan_and_collect, populate_csv_partitions};
     use arrow::datatypes::{DataType, TimeUnit};
-    use std::env;
     use std::error::Error;
     use std::path::PathBuf;
 
diff --git a/datafusion/core/src/lib.rs b/datafusion/core/src/lib.rs
index cc510bc81f1a8..36e0c3ee9c6a5 100644
--- a/datafusion/core/src/lib.rs
+++ b/datafusion/core/src/lib.rs
@@ -872,6 +872,12 @@ doc_comment::doctest!(
     user_guide_configs
 );
 
+#[cfg(doctest)]
+doc_comment::doctest!(
+    "../../../docs/source/user-guide/runtime_configs.md",
+    user_guide_runtime_configs
+);
+
 #[cfg(doctest)]
 doc_comment::doctest!(
     "../../../docs/source/user-guide/crate-configuration.md",
diff --git a/datafusion/core/tests/sql/mod.rs b/datafusion/core/tests/sql/mod.rs
index 579049692e7dc..2a5597b9fb7ee 100644
--- a/datafusion/core/tests/sql/mod.rs
+++ b/datafusion/core/tests/sql/mod.rs
@@ -63,6 +63,7 @@ pub mod create_drop;
 pub mod explain_analyze;
 pub mod joins;
 mod path_partition;
+mod runtime_config;
 pub mod select;
 mod sql_api;
 
diff --git a/datafusion/core/tests/sql/runtime_config.rs b/datafusion/core/tests/sql/runtime_config.rs
new file mode 100644
index 0000000000000..18e07bb61ed94
--- /dev/null
+++ b/datafusion/core/tests/sql/runtime_config.rs
@@ -0,0 +1,166 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Tests for runtime configuration SQL interface
+
+use std::sync::Arc;
+
+use datafusion::execution::context::SessionContext;
+use datafusion::execution::context::TaskContext;
+use datafusion_physical_plan::common::collect;
+
+#[tokio::test]
+async fn test_memory_limit_with_spill() {
+    let ctx = SessionContext::new();
+
+    ctx.sql("SET datafusion.runtime.memory_limit = '1M'")
+        .await
+        .unwrap()
+        .collect()
+        .await
+        .unwrap();
+
+    ctx.sql("SET datafusion.execution.sort_spill_reservation_bytes = 0")
+        .await
+        .unwrap()
+        .collect()
+        .await
+        .unwrap();
+
+    let query = "select * from generate_series(1,10000000) as t1(v1) order by v1;";
+    let df = ctx.sql(query).await.unwrap();
+
+    let plan = df.create_physical_plan().await.unwrap();
+    let task_ctx = Arc::new(TaskContext::from(&ctx.state()));
+    let stream = plan.execute(0, task_ctx).unwrap();
+
+    let _results = collect(stream).await;
+    let metrics = plan.metrics().unwrap();
+    let spill_count = metrics.spill_count().unwrap();
+    assert!(spill_count > 0, "Expected spills but none occurred");
+}
+
+#[tokio::test]
+async fn test_no_spill_with_adequate_memory() {
+    let ctx = SessionContext::new();
+
+    ctx.sql("SET datafusion.runtime.memory_limit = '10M'")
+        .await
+        .unwrap()
+        .collect()
+        .await
+        .unwrap();
+    ctx.sql("SET datafusion.execution.sort_spill_reservation_bytes = 0")
+        .await
+        .unwrap()
+        .collect()
+        .await
+        .unwrap();
+
+    let query = "select * from generate_series(1,100000) as t1(v1) order by v1;";
+    let df = ctx.sql(query).await.unwrap();
+
+    let plan = df.create_physical_plan().await.unwrap();
+    let task_ctx = Arc::new(TaskContext::from(&ctx.state()));
+    let stream = plan.execute(0, task_ctx).unwrap();
+
+    let _results = collect(stream).await;
+    let metrics = plan.metrics().unwrap();
+    let spill_count = metrics.spill_count().unwrap();
+    assert_eq!(spill_count, 0, "Expected no spills but some occurred");
+}
+
+#[tokio::test]
+async fn test_multiple_configs() {
+    let ctx = SessionContext::new();
+
+    ctx.sql("SET datafusion.runtime.memory_limit = '100M'")
+        .await
+        .unwrap()
+        .collect()
+        .await
+        .unwrap();
+    ctx.sql("SET datafusion.execution.batch_size = '2048'")
+        .await
+        .unwrap()
+        .collect()
+        .await
+        .unwrap();
+
+    let query = "select * from generate_series(1,100000) as t1(v1) order by v1;";
+    let result = ctx.sql(query).await.unwrap().collect().await;
+
+    assert!(result.is_ok(), "Should not fail due to memory limit");
+
+    let state = ctx.state();
+    let batch_size = state.config().options().execution.batch_size;
+    assert_eq!(batch_size, 2048);
+}
+
+#[tokio::test]
+async fn test_memory_limit_enforcement() {
+    let ctx = SessionContext::new();
+
+    ctx.sql("SET datafusion.runtime.memory_limit = '1M'")
+        .await
+        .unwrap()
+        .collect()
+        .await
+        .unwrap();
+
+    let query = "select * from generate_series(1,100000) as t1(v1) order by v1;";
+    let result = ctx.sql(query).await.unwrap().collect().await;
+
+    assert!(result.is_err(), "Should fail due to memory limit");
+
+    ctx.sql("SET datafusion.runtime.memory_limit = '100M'")
+        .await
+        .unwrap()
+        .collect()
+        .await
+        .unwrap();
+
+    let result = ctx.sql(query).await.unwrap().collect().await;
+
+    assert!(result.is_ok(), "Should not fail due to memory limit");
+}
+
+#[tokio::test]
+async fn test_invalid_memory_limit() {
+    let ctx = SessionContext::new();
+
+    let result = ctx
+        .sql("SET datafusion.runtime.memory_limit = '100X'")
+        .await;
+
+    assert!(result.is_err());
+    let error_message = result.unwrap_err().to_string();
+    assert!(error_message.contains("Unsupported unit 'X'"));
+}
+
+#[tokio::test]
+async fn test_unknown_runtime_config() {
+    let ctx = SessionContext::new();
+
+    let result = ctx
+        .sql("SET datafusion.runtime.unknown_config = 'value'")
+        .await;
+
+    assert!(result.is_err());
+    let error_message = result.unwrap_err().to_string();
+    assert!(error_message.contains("Unknown runtime configuration"));
+}
diff --git a/datafusion/execution/src/runtime_env.rs b/datafusion/execution/src/runtime_env.rs
index 95f14f485792a..cb085108819eb 100644
--- a/datafusion/execution/src/runtime_env.rs
+++ b/datafusion/execution/src/runtime_env.rs
@@ -27,7 +27,7 @@ use crate::{
 };
 
 use crate::cache::cache_manager::{CacheManager, CacheManagerConfig};
-use datafusion_common::Result;
+use datafusion_common::{config::ConfigEntry, Result};
 use object_store::ObjectStore;
 use std::path::PathBuf;
 use std::sync::Arc;
@@ -268,4 +268,56 @@ impl RuntimeEnvBuilder {
     pub fn build_arc(self) -> Result<Arc<RuntimeEnv>> {
         self.build().map(Arc::new)
     }
+
+    /// Create a new RuntimeEnvBuilder from an existing RuntimeEnv
+    pub fn from_runtime_env(runtime_env: &RuntimeEnv) -> Self {
+        let cache_config = CacheManagerConfig {
+            table_files_statistics_cache: runtime_env
+                .cache_manager
+                .get_file_statistic_cache(),
+            list_files_cache: runtime_env.cache_manager.get_list_files_cache(),
+        };
+
+        Self {
+            disk_manager: DiskManagerConfig::Existing(Arc::clone(
+                &runtime_env.disk_manager,
+            )),
+            memory_pool: Some(Arc::clone(&runtime_env.memory_pool)),
+            cache_manager: cache_config,
+            object_store_registry: Arc::clone(&runtime_env.object_store_registry),
+        }
+    }
+
+    /// Returns a list of all available runtime configurations with their current values and descriptions
+    pub fn entries(&self) -> Vec<ConfigEntry> {
+        // Memory pool configuration
+        vec![ConfigEntry {
+            key: "datafusion.runtime.memory_limit".to_string(),
+            value: None, // Default is system-dependent
+            description: "Maximum memory limit for query execution. Supports suffixes K (kilobytes), M (megabytes), and G (gigabytes). Example: '2G' for 2 gigabytes.",
+        }]
+    }
+
+    /// Generate documentation that can be included in the user guide
+    pub fn generate_config_markdown() -> String {
+        use std::fmt::Write as _;
+
+        let s = Self::default();
+
+        let mut docs = "| key | default | description |\n".to_string();
+        docs += "|-----|---------|-------------|\n";
+        let mut entries = s.entries();
+        entries.sort_unstable_by(|a, b| a.key.cmp(&b.key));
+
+        for entry in &entries {
+            let _ = writeln!(
+                &mut docs,
+                "| {} | {} | {} |",
+                entry.key,
+                entry.value.as_deref().unwrap_or("NULL"),
+                entry.description
+            );
+        }
+        docs
+    }
 }
diff --git a/dev/update_runtime_config_docs.sh b/dev/update_runtime_config_docs.sh
new file mode 100755
index 0000000000000..0d9d0f1033236
--- /dev/null
+++ b/dev/update_runtime_config_docs.sh
@@ -0,0 +1,76 @@
+#!/bin/bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+
+set -e
+
+SOURCE_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+cd "${SOURCE_DIR}/../" && pwd
+
+TARGET_FILE="docs/source/user-guide/runtime_configs.md"
+PRINT_CONFIG_DOCS_COMMAND="cargo run --manifest-path datafusion/core/Cargo.toml --bin print_runtime_config_docs"
+
+echo "Inserting header"
+cat <<'EOF' > "$TARGET_FILE"
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+<!---
+This file was generated by the dev/update_runtime_config_docs.sh script.
+Do not edit it manually as changes will be overwritten.
+Instead, edit dev/update_runtime_config_docs.sh or the docstrings in datafusion/execution/src/runtime_env.rs.
+-->
+
+# Runtime Environment Configurations
+
+DataFusion runtime configurations can be set via SQL using the `SET` command.
+
+For example, to configure `datafusion.runtime.memory_limit`:
+
+```sql
+SET datafusion.runtime.memory_limit = '2G';
+```
+
+The following runtime configuration settings are available:
+
+EOF
+
+echo "Running CLI and inserting runtime config docs table"
+$PRINT_CONFIG_DOCS_COMMAND >> "$TARGET_FILE"
+
+echo "Running prettier"
+npx prettier@2.3.2 --write "$TARGET_FILE"
+
+echo "'$TARGET_FILE' successfully updated!"
diff --git a/docs/source/index.rst b/docs/source/index.rst
index 0dc947fdea579..e920a0f036cbe 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -116,6 +116,7 @@ To get started, see
    user-guide/expressions
    user-guide/sql/index
    user-guide/configs
+   user-guide/runtime_configs
    user-guide/explain-usage
    user-guide/faq
 
diff --git a/docs/source/user-guide/runtime_configs.md b/docs/source/user-guide/runtime_configs.md
new file mode 100644
index 0000000000000..feef709db9929
--- /dev/null
+++ b/docs/source/user-guide/runtime_configs.md
@@ -0,0 +1,40 @@
+<!---
+  Licensed to the Apache Software Foundation (ASF) under one
+  or more contributor license agreements.  See the NOTICE file
+  distributed with this work for additional information
+  regarding copyright ownership.  The ASF licenses this file
+  to you under the Apache License, Version 2.0 (the
+  "License"); you may not use this file except in compliance
+  with the License.  You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+  Unless required by applicable law or agreed to in writing,
+  software distributed under the License is distributed on an
+  "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+  KIND, either express or implied.  See the License for the
+  specific language governing permissions and limitations
+  under the License.
+-->
+
+<!---
+This file was generated by the dev/update_runtime_config_docs.sh script.
+Do not edit it manually as changes will be overwritten.
+Instead, edit dev/update_runtime_config_docs.sh or the docstrings in datafusion/execution/src/runtime_env.rs.
+-->
+
+# Runtime Environment Configurations
+
+DataFusion runtime configurations can be set via SQL using the `SET` command.
+
+For example, to configure `datafusion.runtime.memory_limit`:
+
+```sql
+SET datafusion.runtime.memory_limit = '2G';
+```
+
+The following runtime configuration settings are available:
+
+| key                             | default | description                                                                                                                                 |
+| ------------------------------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------- |
+| datafusion.runtime.memory_limit | NULL    | Maximum memory limit for query execution. Supports suffixes K (kilobytes), M (megabytes), and G (gigabytes). Example: '2G' for 2 gigabytes. |


Reviews:
user: login: 2010YOUY01
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T05:03:05Z
user: login: berkaysynnada
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-08T14:12:30Z
user: login: kumarlokesh
state: COMMENTED
author_association: CONTRIBUTOR
submitted_at: 2025-04-12T17:42:28Z
user: login: 2010YOUY01
body: Thank you. I plan to leave this PR open for 2 days more before merging, in case anyone has additional comments.
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-14T05:23:45Z
user: login: alamb
body: This looks really nice -- thank you @kumarlokesh and @2010YOUY01 
state: APPROVED
author_association: CONTRIBUTOR
submitted_at: 2025-04-15T18:39:51Z

Review comments:
diff_hunk: @@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Tests for runtime configuration SQL interface
+
+use datafusion::prelude::*;
+use datafusion_common::Result;
+
+#[tokio::test]
+async fn test_set_memory_limit() -> Result<()> {
+    let ctx = SessionContext::new();
+
+    // Set memory limit to 100MB using SQL - note the quotes around the value
+    ctx.sql("SET datafusion.runtime.memory_limit = '100M'")
path: datafusion/core/tests/sql/runtime_config.rs
user: login: 2010YOUY01
body: Running this test can't ensure this config is set, because it can run without setting the memory limit and use the default `UnboundedMemoryPool`, instead I think it should assert the spill count for a query that has spilled some intermediate data.
```
SET datafusion.runtime.memory_limit = '1M'
set datafusion.execution.sort_spill_reservation_bytes = 0;

select * from generate_series(1, 100000) as t1(v1) order by v1;
-- And assert spill-count from the query is > 0
```
You can check this https://github.com/apache/datafusion/blob/4ae0edea8eb7d5ad5df07e22a8fb39aa9fd1e2df/datafusion/core/tests/memory_limit/mod.rs#L525 for how to assert the spill file count.

Later after all configurations are added, I think we should make this test case stronger by setting more runtime configs, and do some property test to ensure all of them are properly set.

BTW, I tried the above test locally, and the `generate_series` UDTF seems not registered in the PR branch, but it works in the main branch.
created_at: 2025-04-08T05:03:04Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1036,13 +1040,82 @@ impl SessionContext {
             variable, value, ..
         } = stmt;
 
-        let mut state = self.state.write();
-        state.config_mut().options_mut().set(&variable, &value)?;
-        drop(state);
+        // Check if this is a runtime configuration
+        if variable.starts_with("datafusion.runtime.") {
+            self.set_runtime_variable(&variable, &value)?;
+        } else {
+            let mut state = self.state.write();
+            state.config_mut().options_mut().set(&variable, &value)?;
+            drop(state);
+        }
 
         self.return_empty_dataframe()
     }
 
+    fn set_runtime_variable(&self, variable: &str, value: &str) -> Result<()> {
+        let key = variable.strip_prefix("datafusion.runtime.").unwrap();
+
+        match key {
+            "memory_limit" => {
+                let memory_limit = self.parse_memory_limit(value)?;
+
+                let current_runtime = {
+                    let state = self.state.read();
+                    Arc::<RuntimeEnv>::clone(state.runtime_env())
+                };
+
+                let mut builder = RuntimeEnvBuilder::from_runtime_env(&current_runtime);
+                builder = builder.with_memory_limit(memory_limit, 1.0);
+
+                let new_runtime = builder.build()?;
+                self.update_runtime_env(Arc::new(new_runtime))?;
+            }
+            _ => {
+                return Err(DataFusionError::Plan(format!(
+                    "Unknown runtime configuration: {}",
+                    variable
+                )));
+            }
+        }
+
+        Ok(())
+    }
+
+    /// Parse memory limit from string to number of bytes
+    /// e.g. '1.5G', '100M'
+    fn parse_memory_limit(&self, limit: &str) -> Result<usize> {
path: datafusion/core/src/execution/context/mod.rs
user: login: berkaysynnada
body: Maybe we can open this function to external uses?
created_at: 2025-04-08T14:12:30Z
author_association: CONTRIBUTOR
diff_hunk: @@ -0,0 +1,73 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements.  See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership.  The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License.  You may obtain a copy of the License at
+//
+//   http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied.  See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+//! Tests for runtime configuration SQL interface
+
+use datafusion::prelude::*;
+use datafusion_common::Result;
+
+#[tokio::test]
+async fn test_set_memory_limit() -> Result<()> {
+    let ctx = SessionContext::new();
+
+    // Set memory limit to 100MB using SQL - note the quotes around the value
+    ctx.sql("SET datafusion.runtime.memory_limit = '100M'")
path: datafusion/core/tests/sql/runtime_config.rs
user: login: kumarlokesh
body: > Running this test can't ensure this config is set, because it can run without setting the memory limit and use the default `UnboundedMemoryPool`, instead I think it should assert the spill count for a query that has spilled some intermediate data.
> 
> ```
> SET datafusion.runtime.memory_limit = '1M'
> set datafusion.execution.sort_spill_reservation_bytes = 0;
> 
> select * from generate_series(1, 100000) as t1(v1) order by v1;
> -- And assert spill-count from the query is > 0
> ```
> 
> You can check this
> 
> https://github.com/apache/datafusion/blob/4ae0edea8eb7d5ad5df07e22a8fb39aa9fd1e2df/datafusion/core/tests/memory_limit/mod.rs#L525
> 
> for how to assert the spill file count.
> Later after all configurations are added, I think we should make this test case stronger by setting more runtime configs, and do some property test to ensure all of them are properly set.
> 
> BTW, I tried the above test locally, and the `generate_series` UDTF seems not registered in the PR branch, but it works in the main branch.

@2010YOUY01 @berkaysynnada addressed above in https://github.com/apache/datafusion/pull/15594/commits/6501f218e2c3f7c97316794b2137e7b3672b77a9.
created_at: 2025-04-12T17:42:28Z
author_association: CONTRIBUTOR
diff_hunk: @@ -1036,13 +1040,73 @@ impl SessionContext {
             variable, value, ..
         } = stmt;
 
-        let mut state = self.state.write();
-        state.config_mut().options_mut().set(&variable, &value)?;
-        drop(state);
+        // Check if this is a runtime configuration
+        if variable.starts_with("datafusion.runtime.") {
+            self.set_runtime_variable(&variable, &value)?;
+        } else {
+            let mut state = self.state.write();
+            state.config_mut().options_mut().set(&variable, &value)?;
+            drop(state);
+        }
 
         self.return_empty_dataframe()
     }
 
+    fn set_runtime_variable(&self, variable: &str, value: &str) -> Result<()> {
+        let key = variable.strip_prefix("datafusion.runtime.").unwrap();
+
+        match key {
+            "memory_limit" => {
+                let memory_limit = Self::parse_memory_limit(value)?;
+
+                let mut state = self.state.write();
+                let mut builder =
+                    RuntimeEnvBuilder::from_runtime_env(state.runtime_env());
path: datafusion/core/src/execution/context/mod.rs
user: login: 2010YOUY01
body: This implementation is all cloning `Arc`s inside `RuntimeEnv`, I think it shouldn't have any issue.
created_at: 2025-04-14T05:22:45Z
author_association: CONTRIBUTOR

PR comments:
user: login: berkaysynnada
created_at: 2025-04-06T12:28:18Z
body: Hello @kumarlokesh. Thank you for working on this. I have 2 questions/concerns. Let's discuss on them a bit to get a future-proof design

1) There are also `runtime_env: Arc<RuntimeEnv>` in `SessionState` and `TaskContext` structs, and this `RuntimeEnv` has a field of `pub memory_pool: Arc<dyn MemoryPool>`. These MemoryPool implementations have some configurations about memory limit as well. Keeping 2 different things for the same purposes always bring trouble. So, I wonder can we somehow connect those 2 points.

2) Do you plan to practice this configuration in any part of the execution? While bringing such a feature, implementing at least one use-case makes things more clear and becomes a guidance for the further developments.
author_association: CONTRIBUTOR
user: login: kumarlokesh
created_at: 2025-04-07T20:08:08Z
body: > Hello @kumarlokesh. Thank you for working on this. I have 2 questions/concerns. Let's discuss on them a bit to get a future-proof design
> 
> 1. There are also `runtime_env: Arc<RuntimeEnv>` in `SessionState` and `TaskContext` structs, and this `RuntimeEnv` has a field of `pub memory_pool: Arc<dyn MemoryPool>`. These MemoryPool implementations have some configurations about memory limit as well. Keeping 2 different things for the same purposes always bring trouble. So, I wonder can we somehow connect those 2 points.
> 2. Do you plan to practice this configuration in any part of the execution? While bringing such a feature, implementing at least one use-case makes things more clear and becomes a guidance for the further developments.

@berkaysynnada Thank you for the feedback! 

1. Have revised the PR to ensure runtime environment configuration like memory-limit is exposed through the already defined RuntimeEnv -> memory_pool setting.

2. As a user can set the configuration `datafusion.runtime.memory_limit` (and other available runtime environment configurations) through SQL interface, I think having tests around this behaviour should be a good starting point. Can't think of another good use case besides this. Thoughts?
author_association: CONTRIBUTOR
user: login: berkaysynnada
created_at: 2025-04-08T14:17:16Z
body: Thank you @kumarlokesh for addressing my comments. I don't have further suggestions or concern other than @2010YOUY01 shared
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-04-15T18:38:42Z
body: Thanks -- I recommend we merge this after the 47.0.0 release is created (likely tomorrow)
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-04-16T19:32:05Z
body: We made `branch-47` so let's get this one in!
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-04-16T19:32:26Z
body: I merged up from main to rerun CI to be sure all tests pass
author_association: CONTRIBUTOR
user: login: alamb
created_at: 2025-04-17T20:13:01Z
body: 🚀  so exciting -- thank you
author_association: CONTRIBUTOR