title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/chdb/datastore/index.md)

DataStore is chDB's pandas-compatible API that combines the familiar pandas DataFrame interface with the power of SQL query optimization and allows you to write pandas-style code while getting ClickHouse performance.

## Key features[​](#key-features "Direct link to Key features")

- **Pandas Compatibility**: 209 pandas DataFrame methods, 56 `.str` methods, 42+ `.dt` methods
- **SQL Optimization**: Operations automatically compile to optimized SQL queries
- **Lazy Evaluation**: Operations are deferred until results are needed
- **630+ API Methods**: Comprehensive API surface for data manipulation
- **ClickHouse Extensions**: Additional accessors (`.arr`, `.json`, `.url`, `.ip`, `.geo`) not available in pandas

## Architecture[​](#architecture "Direct link to Architecture")

DataStore uses **lazy evaluation** with **dual-engine execution**:

1. **Lazy Operation Chain**: Operations are recorded, not executed immediately
2. **Smart Engine Selection**: QueryPlanner routes each segment to optimal engine (chDB for SQL, Pandas for complex ops)
3. **Intermediate Caching**: Results cached at each step for fast iterative exploration

See [Execution Model](/docs/chdb/datastore/execution-model) for details.

## One-Line migration from Pandas[​](#migration "Direct link to One-Line migration from Pandas")

```
# Before (pandas)
import pandas as pd
df = pd.read_csv("data.csv")
result = df[df['age'] > 25].groupby('city')['salary'].mean()

# After (DataStore) - just change the import!
from chdb import datastore as pd
df = pd.read_csv("data.csv")
result = df[df['age'] > 25].groupby('city')['salary'].mean()
```

Your existing pandas code works unchanged, but now runs on the ClickHouse engine.

## Performance comparison[​](#performance "Direct link to Performance comparison")

DataStore delivers significant performance improvements over pandas, especially for aggregation and complex pipelines:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Operation Pandas DataStore Speedup|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | GroupBy count 347ms 17ms **19.93x**| Complex pipeline 2,047ms 380ms **5.39x**| Filter+Sort+Head 1,537ms 350ms **4.40x**| GroupBy agg 406ms 141ms **2.88x** | | | | | | | | | | | | | | | | | | | |

*Benchmark on 10M rows. See [benchmark script](https://github.com/chdb-io/chdb/blob/main/refs/benchmark_datastore_vs_pandas.py) and [Performance Guide](/docs/chdb/guides/pandas-performance) for details.*

## When to use DataStore[​](#when-to-use "Direct link to When to use DataStore")

**Use DataStore when:**

- Working with large datasets (millions of rows)
- Performing aggregations and groupby operations
- Querying data from files, databases, or cloud storage
- Building complex data pipelines
- You want pandas API with better performance

**Use raw SQL API when:**

- You prefer writing SQL directly
- You need fine-grained control over query execution
- Working with ClickHouse-specific features not exposed in pandas API

## Feature comparison[​](#comparison "Direct link to Feature comparison")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Feature Pandas Polars DuckDB DataStore|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Pandas API compatible - Partial No **Full**| Lazy evaluation No Yes Yes **Yes**| SQL query support No Yes Yes **Yes**| ClickHouse functions No No No **Yes**| String/DateTime accessors Yes Yes No **Yes + extras**| Array/JSON/URL/IP/Geo No Partial No **Yes**| Direct file queries No Yes Yes **Yes**| Cloud storage support No Limited Yes **Yes** | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

## API statistics[​](#api-stats "Direct link to API statistics")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Category Count Coverage|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | DataFrame methods 209 100% of pandas|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Series.str accessor 56 100% of pandas|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Series.dt accessor 42+ 100%+ (includes ClickHouse extras)|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Series.arr accessor 37 ClickHouse-specific|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Series.json accessor 13 ClickHouse-specific|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Series.url accessor 15 ClickHouse-specific|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | Series.ip accessor 9 ClickHouse-specific|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | Series.geo accessor 14 ClickHouse-specific|  |  |  | | --- | --- | --- | | **Total API methods** **630+** - | | | | | | | | | | | | | | | | | | | | | | | | | | | | | |

## Documentation navigation[​](#navigation "Direct link to Documentation navigation")

### Getting Started[​](#getting-started "Direct link to Getting Started")

- [Quickstart](/docs/chdb/datastore/quickstart) - Installation and basic usage
- [Migration from Pandas](/docs/chdb/guides/migration-from-pandas) - Step-by-step migration guide

### API reference[​](#api-reference "Direct link to API reference")

- [Factory Methods](/docs/chdb/datastore/factory-methods) - Creating DataStore from various sources
- [Query Building](/docs/chdb/datastore/query-building) - SQL-style query operations
- [Pandas Compatibility](/docs/chdb/datastore/pandas-compat) - All 209 pandas-compatible methods
- [Accessors](/docs/chdb/datastore/accessors) - String, DateTime, Array, JSON, URL, IP, Geo accessors
- [Aggregation](/docs/chdb/datastore/aggregation) - Aggregate and window functions
- [I/O Operations](/docs/chdb/datastore/io) - Reading and writing data

### Advanced topics[​](#advanced-topics "Direct link to Advanced topics")

- [Execution Model](/docs/chdb/datastore/execution-model) - Lazy evaluation and caching
- [Class Reference](/docs/chdb/datastore/class-reference) - Complete API reference

### Configuration & debugging[​](#configuration-debugging "Direct link to Configuration & debugging")