title_wip

### Recommendations[​](#recommendations "Direct link to Recommendations")

1. **For analytics workloads:** Use Parquet

   - Columnar format allows reading only needed columns
   - Excellent compression
   - Preserves data types
2. **For data exchange:** Use CSV or JSON

   - Universal compatibility
   - Human-readable
3. **For pandas interop:** Use Feather or Arrow

   - Fastest serialization
   - Type preservation

---

## Compression Support[​](#compression "Direct link to Compression Support")

### Reading Compressed Files[​](#read-compressed "Direct link to Reading Compressed Files")

```
# Auto-detect from extension
ds = pd.read_csv("data.csv.gz")
ds = pd.read_csv("data.csv.bz2")
ds = pd.read_csv("data.csv.xz")
ds = pd.read_csv("data.csv.zst")

# Explicit compression
ds = pd.read_csv("data.csv", compression='gzip')
```

### Writing Compressed Files[​](#write-compressed "Direct link to Writing Compressed Files")

```
# CSV with compression
ds.to_csv("output.csv.gz", compression='gzip')
ds.to_csv("output.csv.bz2", compression='bz2')

# Parquet (always compressed)
ds.to_parquet("output.parquet", compression='snappy')  # Default
ds.to_parquet("output.parquet", compression='gzip')
ds.to_parquet("output.parquet", compression='zstd')    # Best ratio
ds.to_parquet("output.parquet", compression='lz4')     # Fastest
```

### Compression Options[​](#compression-options "Direct link to Compression Options")

|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Compression Speed Ratio Use Case|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `snappy` Very Fast Low Default for Parquet|  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `lz4` Very Fast Low Speed priority|  |  |  |  |  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `gzip` Medium High Compatibility|  |  |  |  |  |  |  |  | | --- | --- | --- | --- | --- | --- | --- | --- | | `zstd` Fast Very High Best balance|  |  |  |  | | --- | --- | --- | --- | | `bz2` Slow Very High Maximum compression | | | | | | | | | | | | | | | | | | | | | | | |

---

## Streaming I/O[​](#streaming "Direct link to Streaming I/O")

For very large files that don't fit in memory:

### Chunked Reading[​](#chunked-read "Direct link to Chunked Reading")

```
# Read in chunks
for chunk in pd.read_csv("large.csv", chunksize=100000):
    # Process each chunk
    process(chunk)

# Using iterator
reader = pd.read_csv("large.csv", iterator=True)
chunk = reader.get_chunk(10000)
```

### Using ClickHouse Streaming[​](#clickhouse-streaming "Direct link to Using ClickHouse Streaming")

```
from chdb.datastore import DataStore

# Stream from file without loading all into memory
ds = DataStore.from_file("huge.parquet")

# Operations are lazy - only computes what's needed
result = ds.filter(ds['amount'] > 1000).head(100)
```

---

## Remote Data Sources[​](#remote "Direct link to Remote Data Sources")

### HTTP/HTTPS[​](#http "Direct link to HTTP/HTTPS")

```
# Read from URL
ds = pd.read_csv("https://example.com/data.csv")
ds = pd.read_parquet("https://example.com/data.parquet")
```

### S3[​](#s3 "Direct link to S3")

```
from chdb.datastore import DataStore

# Anonymous access
ds = DataStore.uri("s3://bucket/data.parquet?nosign=true")

# With credentials
ds = DataStore.from_s3(
    "s3://bucket/data.parquet",
    access_key_id="KEY",
    secret_access_key="SECRET"
)
```

### GCS, Azure, HDFS[​](#cloud "Direct link to GCS, Azure, HDFS")

See [Factory Methods](/docs/chdb/datastore/factory-methods) for cloud storage options.

---

## Best Practices[​](#best-practices "Direct link to Best Practices")

### 1. Use Parquet for Large Files[​](#use-parquet-for-large-files "Direct link to 1. Use Parquet for Large Files")

```
# Convert CSV to Parquet for better performance
ds = pd.read_csv("large.csv")
ds.to_parquet("large.parquet")

# Future reads are much faster
ds = pd.read_parquet("large.parquet")
```

### 2. Select Only Needed Columns[​](#select-only-needed-columns "Direct link to 2. Select Only Needed Columns")

```
# Efficient - only reads col1 and col2
ds = pd.read_parquet("data.parquet", columns=['col1', 'col2'])

# Inefficient - reads all columns then filters
ds = pd.read_parquet("data.parquet")[['col1', 'col2']]
```

### 3. Use Compression[​](#use-compression "Direct link to 3. Use Compression")

```
# Smaller file size, usually faster due to less I/O
ds.to_parquet("output.parquet", compression='zstd')
```

### 4. Batch Writes[​](#batch-writes "Direct link to 4. Batch Writes")

```
# Write once, not in a loop
result = process_all_data(ds)
result.to_parquet("output.parquet")

# NOT this (inefficient)
for chunk in chunks:
    chunk.to_parquet(f"output_{i}.parquet")
```

- [Reading Data](#reading)
  - [CSV Files](#read-csv)- [Parquet Files](#read-parquet)- [JSON Files](#read-json)- [Excel Files](#read-excel)- [SQL Databases](#read-sql)- [Other Formats](#read-other)- [Writing Data](#writing)
    - [to\_csv](#to-csv)- [to\_parquet](#to-parquet)- [to\_json](#to-json)- [to\_excel](#to-excel)- [to\_sql](#to-sql-method)- [Other Export Methods](#to-other)- [File Format Comparison](#format-comparison)
      - [Recommendations](#recommendations)- [Compression Support](#compression)
        - [Reading Compressed Files](#read-compressed)- [Writing Compressed Files](#write-compressed)- [Compression Options](#compression-options)- [Streaming I/O](#streaming)
          - [Chunked Reading](#chunked-read)- [Using ClickHouse Streaming](#clickhouse-streaming)- [Remote Data Sources](#remote)
            - [HTTP/HTTPS](#http)- [S3](#s3)- [GCS, Azure, HDFS](#cloud)- [Best Practices](#best-practices)
              - [1. Use Parquet for Large Files](#use-parquet-for-large-files)- [2. Select Only Needed Columns](#select-only-needed-columns)- [3. Use Compression](#use-compression)- [4. Batch Writes](#batch-writes)

Was this page helpful?