title_wip

### DataStore[​](#datastore-row-order "Direct link to DataStore")

Row order is **automatically preserved** for most operations:

```
ds = pd.read_csv("data.csv")
print(ds.head())  # Matches file order

# Filter preserves order
ds_filtered = ds[ds['age'] > 25]  # Same order as pandas
```

DataStore automatically tracks original row positions internally (using `rowNumberInAllBlocks()`) to ensure order consistency with pandas.

### When Order Is Preserved[​](#order-preserved "Direct link to When Order Is Preserved")

- File sources (CSV, Parquet, JSON, etc.)
- pandas DataFrame sources
- Filter operations
- Column selection
- After explicit `sort()` or `sort_values()`
- Operations that define order (`nlargest()`, `nsmallest()`, `head()`, `tail()`)

### When Order May Differ[​](#order-may-differ "Direct link to When Order May Differ")

- After `groupby()` aggregations (use `sort_values()` to ensure consistent order)
- After `merge()` / `join()` with certain join types
- In **performance mode** (`config.use_performance_mode()`): row order is not guaranteed for any operation. See [Performance Mode](/docs/chdb/configuration/performance-mode).

---

## 5. No inplace Parameter[​](#no-inplace "Direct link to 5. No inplace Parameter")

### pandas[​](#pandas-inplace "Direct link to pandas")

```
df.drop(columns=['col'], inplace=True)  # Modifies df
df.fillna(0, inplace=True)              # Modifies df
df.rename(columns={'old': 'new'}, inplace=True)
```

### DataStore[​](#datastore-inplace "Direct link to DataStore")

`inplace=True` is not supported. Always assign the result:

```
ds = ds.drop(columns=['col'])           # Returns new DataStore
ds = ds.fillna(0)                       # Returns new DataStore
ds = ds.rename(columns={'old': 'new'})  # Returns new DataStore
```

### Why No inplace?[​](#why-no-inplace "Direct link to Why No inplace?")

DataStore uses immutable operations to enable:

- Query building (lazy evaluation)
- Thread safety
- Easier debugging
- Cleaner code

---

## 6. Index Support[​](#index "Direct link to 6. Index Support")

### pandas[​](#pandas-index "Direct link to pandas")

Full index support:

```
df = df.set_index('id')
df.loc['user123']           # Label-based access
df.loc['a':'z']             # Label-based slicing
df.reset_index()
df.index.name = 'user_id'
```

### DataStore[​](#datastore-index "Direct link to DataStore")

Simplified index support:

```
# Basic operations work
ds.loc[0:10]               # Integer position
ds.iloc[0:10]              # Same as loc for DataStore

# For pandas-style index operations, convert first
df = ds.to_df()
df = df.set_index('id')
df.loc['user123']
```

### DataStore Source Matters[​](#datastore-source-matters "Direct link to DataStore Source Matters")

- **DataFrame source**: Preserves pandas index
- **File source**: Uses simple integer index

---

## 7. Comparison Behavior[​](#comparison "Direct link to 7. Comparison Behavior")

### Comparing with pandas[​](#comparing-with-pandas "Direct link to Comparing with pandas")

pandas doesn't recognize DataStore objects:

```
import pandas as pd
from chdb import datastore as ds

pdf = pd.DataFrame({'a': [1, 2, 3]})
dsf = ds.DataFrame({'a': [1, 2, 3]})

# This doesn't work as expected
pdf == dsf  # pandas doesn't know DataStore

# Solution: convert DataStore to pandas
pdf.equals(dsf.to_pandas())  # True
```

### Using equals()[​](#using-equals "Direct link to Using equals()")

```
# DataStore.equals() also works
dsf.equals(pdf)  # Compares with pandas DataFrame
```

---

## 8. Type Inference[​](#types "Direct link to 8. Type Inference")

### pandas[​](#pandas-types "Direct link to pandas")

Uses numpy/pandas types:

```
df['col'].dtype  # int64, float64, object, datetime64, etc.
```

### DataStore[​](#datastore-types "Direct link to DataStore")

May use ClickHouse types:

```
ds['col'].dtype  # Int64, Float64, String, DateTime, etc.

# Types are converted when going to pandas
df = ds.to_df()
df['col'].dtype  # Now pandas type
```

### Explicit Casting[​](#explicit-casting "Direct link to Explicit Casting")

```
# Force specific type
ds['col'] = ds['col'].astype('int64')
```

---

## 9. Memory Model[​](#memory "Direct link to 9. Memory Model")

### pandas[​](#pandas-memory "Direct link to pandas")

All data lives in memory:

```
df = pd.read_csv("huge.csv")  # 10GB in memory!
```

### DataStore[​](#datastore-memory "Direct link to DataStore")

Data stays at source until needed:

```
ds = pd.read_csv("huge.csv")  # Just metadata
ds = ds.filter(ds['year'] == 2024)  # Still just metadata

# Only filtered result is loaded
df = ds.to_df()  # Maybe only 1GB now
```

---

## 10. Error Messages[​](#errors "Direct link to 10. Error Messages")

### Different Error Sources[​](#different-error-sources "Direct link to Different Error Sources")

- **pandas errors**: From pandas library
- **DataStore errors**: From chDB or ClickHouse

```
# May see ClickHouse-style errors
# "Code: 62. DB::Exception: Syntax error..."
```

### Debugging Tips[​](#debugging-tips "Direct link to Debugging Tips")

```
# View the SQL to debug
print(ds.to_sql())

# See execution plan
ds.explain()

# Enable debug logging
from chdb.datastore.config import config
config.enable_debug()
```

---

## Migration Checklist[​](#checklist "Direct link to Migration Checklist")

When migrating from pandas:

- Change import statement
- Remove `inplace=True` parameters
- Add explicit `to_df()` where pandas DataFrame is required
- Add sorting if row order matters
- Use `to_pandas()` for comparison tests
- Test with representative data sizes

---

## Quick Reference[​](#quick-ref "Direct link to Quick Reference")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| pandas DataStore|  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `df[condition]` Same (returns DataStore)|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `df.groupby()` Same (returns LazyGroupBy)|  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `df.drop(inplace=True)` `ds = ds.drop()`| `df.equals(other)` `ds.to_pandas().equals(other)`| `df.loc['label']` `ds.to_df().loc['label']`| `print(df)` Same (triggers execution)|  |  | | --- | --- | | `len(df)` Same (triggers execution) | | | | | | | | | | | | | | | |

- [Summary Table](#summary)- [1. Lazy vs Eager Execution](#lazy-execution)
    - [pandas (Eager)](#pandas-eager)- [DataStore (Lazy)](#datastore-lazy)- [Why It Matters](#why-lazy)- [2. Return Types](#return-types)
      - [pandas](#pandas-return-types)- [DataStore](#datastore-return-types)- [Converting to pandas Types](#converting-to-pandas-types)- [3. Execution Triggers](#triggers)
        - [Operations That Stay Lazy](#stay-lazy)- [4. Row Order](#row-order)
          - [pandas](#pandas-row-order)- [DataStore](#datastore-row-order)- [When Order Is Preserved](#order-preserved)- [When Order May Differ](#order-may-differ)- [5. No inplace Parameter](#no-inplace)
            - [pandas](#pandas-inplace)- [DataStore](#datastore-inplace)- [Why No inplace?](#why-no-inplace)- [6. Index Support](#index)
              - [pandas](#pandas-index)- [DataStore](#datastore-index)- [DataStore Source Matters](#datastore-source-matters)- [7. Comparison Behavior](#comparison)
                - [Comparing with pandas](#comparing-with-pandas)- [Using equals()](#using-equals)- [8. Type Inference](#types)
                  - [pandas](#pandas-types)- [DataStore](#datastore-types)- [Explicit Casting](#explicit-casting)- [9. Memory Model](#memory)
                    - [pandas](#pandas-memory)- [DataStore](#datastore-memory)- [10. Error Messages](#errors)
                      - [Different Error Sources](#different-error-sources)- [Debugging Tips](#debugging-tips)- [Migration Checklist](#checklist)- [Quick Reference](#quick-ref)

Was this page helpful?