Skip to main content

__ Back to top

__ ` Ctrl ` \+ ` K `

[ ![pandas 2.2.1 documentation - Home](../_static/pandas.svg) ](../index.html)

Site Navigation

  * [ Getting started ](../getting_started/index.html)
  * [ User Guide ](index.html)
  * [ API reference ](../reference/index.html)
  * [ Development ](../development/index.html)
  * [ Release notes ](../whatsnew/index.html)

  * [ __ GitHub  ](https://github.com/pandas-dev/pandas "GitHub")
  * [ __ Twitter  ](https://twitter.com/pandas_dev "Twitter")
  * [ __ Mastodon  ](https://fosstodon.org/@pandas_dev "Mastodon")

Site Navigation

  * [ Getting started ](../getting_started/index.html)
  * [ User Guide ](index.html)
  * [ API reference ](../reference/index.html)
  * [ Development ](../development/index.html)
  * [ Release notes ](../whatsnew/index.html)

  * [ __ GitHub  ](https://github.com/pandas-dev/pandas "GitHub")
  * [ __ Twitter  ](https://twitter.com/pandas_dev "Twitter")
  * [ __ Mastodon  ](https://fosstodon.org/@pandas_dev "Mastodon")

  * [ 10 minutes to pandas ](10min.html)
  * [ Intro to data structures ](dsintro.html)
  * [ Essential basic functionality ](basics.html)
  * [ IO tools (text, CSV, HDF5, â¦) ](io.html)
  * [ PyArrow Functionality ](pyarrow.html)
  * [ Indexing and selecting data ](indexing.html)
  * [ MultiIndex / advanced indexing ](advanced.html)
  * [ Copy-on-Write (CoW) ](copy_on_write.html)
  * [ Merge, join, concatenate and compare ](merging.html)
  * [ Reshaping and pivot tables ](reshaping.html)
  * [ Working with text data ](text.html)
  * [ Working with missing data ](missing_data.html)
  * [ Duplicate Labels ](duplicates.html)
  * [ Categorical data ](categorical.html)
  * [ Nullable integer data type ](integer_na.html)
  * [ Nullable Boolean data type ](boolean.html)
  * [ Chart visualization ](visualization.html)
  * [ Table Visualization ](style.html)
  * [ Group by: split-apply-combine ](groupby.html)
  * [ Windowing operations ](window.html)
  * [ Time series / date functionality ](timeseries.html)
  * [ Time deltas ](timedeltas.html)
  * [ Options and settings ](options.html)
  * [ Enhancing performance ](enhancingperf.html)
  * Scaling to large datasets 
  * [ Sparse data structures ](sparse.html)
  * [ Frequently Asked Questions (FAQ) ](gotchas.html)
  * [ Cookbook ](cookbook.html)

  * [ __ ](../index.html)
  * [ User Guide ](index.html)
  * Scaling to... 

#  Scaling to large datasets  #

pandas provides data structures for in-memory analytics, which makes using
pandas to analyze datasets that are larger than memory datasets somewhat
tricky. Even datasets that are a sizable fraction of memory become unwieldy,
as some pandas operations need to make intermediate copies.

This document provides a few recommendations for scaling your analysis to
larger datasets. Itâs a complement to [ Enhancing performance
](enhancingperf.html#enhancingperf) , which focuses on speeding up analysis
for datasets that fit in memory.

##  Load less data  #

Suppose our raw dataset on disk has many columns.

    
    
    In [1]: import pandas as pd
    
    In [2]: import numpy as np
    
    In [3]: def make_timeseries(start="2000-01-01", end="2000-12-31", freq="1D", seed=None):
       ...:     index = pd.date_range(start=start, end=end, freq=freq, name="timestamp")
       ...:     n = len(index)
       ...:     state = np.random.RandomState(seed)
       ...:     columns = {
       ...:         "name": state.choice(["Alice", "Bob", "Charlie"], size=n),
       ...:         "id": state.poisson(1000, size=n),
       ...:         "x": state.rand(n) * 2 - 1,
       ...:         "y": state.rand(n) * 2 - 1,
       ...:     }
       ...:     df = pd.DataFrame(columns, index=index, columns=sorted(columns))
       ...:     if df.index[-1] == end:
       ...:         df = df.iloc[:-1]
       ...:     return df
       ...: 
    
    In [4]: timeseries = [
       ...:     make_timeseries(freq="1min", seed=i).rename(columns=lambda x: f"{x}_{i}")
       ..