title_wip

[Edit this page](https://github.com/ClickHouse/clickhouse-docs/blob/main/docs/chdb/guides/migration-from-pandas.md)

This guide helps you migrate existing pandas code to DataStore for better performance while maintaining compatibility.

## The One-Line Migration[​](#one-line "Direct link to The One-Line Migration")

The simplest migration is changing your import:

```
# Before (pandas)
import pandas as pd

# After (DataStore)
from chdb import datastore as pd
```

That's it! Most pandas code works unchanged.

## Step-by-Step Migration[​](#step-by-step "Direct link to Step-by-Step Migration")

Install chDB

### Install chDB[​](#step-1 "Direct link to Install chDB")

```
pip install "chdb>=4.0"
```

Change the Import

### Change the Import[​](#step-2 "Direct link to Change the Import")

```
# Change this:
import pandas as pd

# To this:
from chdb import datastore as pd
```

Test Your Code

### Test Your Code[​](#step-3 "Direct link to Test Your Code")

Run your existing code. Most operations work unchanged:

```
from chdb import datastore as pd

# These all work the same
df = pd.read_csv("data.csv")
result = df[df['age'] > 25]
grouped = df.groupby('city')['salary'].mean()
df.to_csv("output.csv")
```

Handle Any Differences

### Handle Any Differences[​](#step-4 "Direct link to Handle Any Differences")

A few operations behave differently. See [Key Differences](#differences) below.

---

## What Works Unchanged[​](#works-unchanged "Direct link to What Works Unchanged")

### Data Loading[​](#loading-unchanged "Direct link to Data Loading")

```
# All these work the same
df = pd.read_csv("data.csv")
df = pd.read_parquet("data.parquet")
df = pd.read_json("data.json")
df = pd.read_excel("data.xlsx")
```

### Filtering[​](#filtering-unchanged "Direct link to Filtering")

```
# Boolean indexing
df[df['age'] > 25]
df[(df['age'] > 25) & (df['city'] == 'NYC')]

# query() method
df.query('age > 25 and salary > 50000')
```

### Selection[​](#selection-unchanged "Direct link to Selection")

```
# Column selection
df['name']
df[['name', 'age']]

# Row selection
df.head(10)
df.tail(10)
df.iloc[0:100]
```

### GroupBy and Aggregation[​](#groupby-unchanged "Direct link to GroupBy and Aggregation")

```
# GroupBy
df.groupby('city')['salary'].mean()
df.groupby(['city', 'dept']).agg({'salary': ['sum', 'mean']})
```

### Sorting[​](#sorting-unchanged "Direct link to Sorting")

```
df.sort_values('salary', ascending=False)
df.sort_values(['city', 'age'])
```

### String Operations[​](#string-unchanged "Direct link to String Operations")

```
df['name'].str.upper()
df['name'].str.contains('John')
df['name'].str.len()
```

### DateTime Operations[​](#datetime-unchanged "Direct link to DateTime Operations")

```
df['date'].dt.year
df['date'].dt.month
df['date'].dt.dayofweek
```

### I/O Operations[​](#io-unchanged "Direct link to I/O Operations")

```
df.to_csv("output.csv")
df.to_parquet("output.parquet")
df.to_json("output.json")
```

---

## Key Differences[​](#differences "Direct link to Key Differences")

### 1. Lazy Evaluation[​](#lazy "Direct link to 1. Lazy Evaluation")

DataStore operations are lazy - they don't execute until results are needed.

**pandas:**

```
# Executes immediately
result = df[df['age'] > 25]
print(type(result))  # pandas.DataFrame
```

**DataStore:**

```
# Builds query, doesn't execute yet
result = ds[ds['age'] > 25]
print(type(result))  # DataStore (lazy)

# Executes when you need the data
print(result)        # Triggers execution
df = result.to_df()  # Triggers execution
```

### 2. Return Types[​](#return-types "Direct link to 2. Return Types")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Operation pandas Returns DataStore Returns|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `df['col']` Series ColumnExpr (lazy)|  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `df[['a', 'b']]` DataFrame DataStore (lazy)|  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | | `df[condition]` DataFrame DataStore (lazy)|  |  |  | | --- | --- | --- | | `df.groupby('x')` GroupBy LazyGroupBy | | | | | | | | | | | | | | |

### 3. No inplace Parameter[​](#no-inplace "Direct link to 3. No inplace Parameter")

DataStore doesn't support `inplace=True`. Always use the return value:

**pandas:**

```
df.drop(columns=['col'], inplace=True)
```

**DataStore:**

```
ds = ds.drop(columns=['col'])  # Assign the result
```

### 4. Comparing DataStores[​](#comparing "Direct link to 4. Comparing DataStores")

pandas doesn't recognize DataStore objects, so use `to_pandas()` for comparison:

```
# This may not work as expected
df == ds  # pandas doesn't know DataStore

# Do this instead
df.equals(ds.to_pandas())
```

### 5. Row Order[​](#row-order "Direct link to 5. Row Order")

DataStore may not preserve row order for file sources (like SQL databases). Use explicit sorting:

```
# pandas preserves order
df = pd.read_csv("data.csv")

# DataStore - use sort for guaranteed order
ds = pd.read_csv("data.csv")
ds = ds.sort('id')  # Explicit ordering
```

---

## Migration Patterns[​](#patterns "Direct link to Migration Patterns")

### Pattern 1: Read-Analyze-Write[​](#pattern-1 "Direct link to Pattern 1: Read-Analyze-Write")

```
# pandas
import pandas as pd
df = pd.read_csv("data.csv")
result = df[df['amount'] > 100].groupby('category')['amount'].sum()
result.to_csv("output.csv")

# DataStore - same code works!
from chdb import datastore as pd
df = pd.read_csv("data.csv")
result = df[df['amount'] > 100].groupby('category')['amount'].sum()
result.to_csv("output.csv")
```

### Pattern 2: DataFrame with pandas Operations[​](#pattern-2 "Direct link to Pattern 2: DataFrame with pandas Operations")

If you need pandas-specific features, convert at the end:

```
from chdb import datastore as pd

# Fast DataStore operations
ds = pd.read_csv("large_data.csv")
ds = ds.filter(ds['date'] >= '2024-01-01')
ds = ds.filter(ds['amount'] > 100)

# Convert to pandas for specific features
df = ds.to_df()
df_pivoted = df.pivot_table(...)  # pandas-specific
```

### Pattern 3: Mixed Workflow[​](#pattern-3 "Direct link to Pattern 3: Mixed Workflow")

```
from chdb import datastore as pd
import pandas

# Start with DataStore for fast filtering
ds = pd.read_csv("huge_file.csv")  # 10M rows
ds = ds.filter(ds['year'] == 2024)  # Fast SQL filter
ds = ds.select('col1', 'col2', 'col3')  # Column pruning

# Convert for pandas-specific operations
df = ds.to_df()  # Now only ~100K rows
result = df.apply(complex_custom_function)  # pandas
```

---

## Performance Comparison[​](#performance "Direct link to Performance Comparison")

DataStore is significantly faster for large datasets:

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Operation pandas DataStore Speedup|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | GroupBy count 347ms 17ms **19.93x**| Complex pipeline 2,047ms 380ms **5.39x**| Filter+Sort+Head 1,537ms 350ms **4.40x**| GroupBy agg 406ms 141ms **2.88x** | | | | | | | | | | | | | | | | | | | |

*Benchmark on 10M rows*

---

## Troubleshooting Migration[​](#troubleshooting "Direct link to Troubleshooting Migration")

### Issue: Operation Not Working[​](#issue-op "Direct link to Issue: Operation Not Working")

Some pandas operations may not be supported. Check:

1. Is the operation in the [compatibility list](/docs/chdb/datastore/pandas-compat)?
2. Try converting to pandas first: `ds.to_df().operation()`

### Issue: Different Results[​](#issue-results "Direct link to Issue: Different Results")

Enable debug logging to understand what's happening:

```
from chdb.datastore.config import config
config.enable_debug()

# View the SQL being generated
ds.filter(ds['x'] > 10).explain()
```