Skip to main content

__ Back to top

__ ` Ctrl ` \+ ` K `

[ ![pandas 2.2.1 documentation - Home](../_static/pandas.svg) ](../index.html)

Site Navigation

  * [ Getting started ](../getting_started/index.html)
  * [ User Guide ](index.html)
  * [ API reference ](../reference/index.html)
  * [ Development ](../development/index.html)
  * [ Release notes ](../whatsnew/index.html)

  * [ __ GitHub  ](https://github.com/pandas-dev/pandas "GitHub")
  * [ __ Twitter  ](https://twitter.com/pandas_dev "Twitter")
  * [ __ Mastodon  ](https://fosstodon.org/@pandas_dev "Mastodon")

Site Navigation

  * [ Getting started ](../getting_started/index.html)
  * [ User Guide ](index.html)
  * [ API reference ](../reference/index.html)
  * [ Development ](../development/index.html)
  * [ Release notes ](../whatsnew/index.html)

  * [ __ GitHub  ](https://github.com/pandas-dev/pandas "GitHub")
  * [ __ Twitter  ](https://twitter.com/pandas_dev "Twitter")
  * [ __ Mastodon  ](https://fosstodon.org/@pandas_dev "Mastodon")

  * [ 10 minutes to pandas ](10min.html)
  * [ Intro to data structures ](dsintro.html)
  * [ Essential basic functionality ](basics.html)
  * [ IO tools (text, CSV, HDF5, â¦) ](io.html)
  * [ PyArrow Functionality ](pyarrow.html)
  * [ Indexing and selecting data ](indexing.html)
  * [ MultiIndex / advanced indexing ](advanced.html)
  * [ Copy-on-Write (CoW) ](copy_on_write.html)
  * [ Merge, join, concatenate and compare ](merging.html)
  * [ Reshaping and pivot tables ](reshaping.html)
  * [ Working with text data ](text.html)
  * [ Working with missing data ](missing_data.html)
  * [ Duplicate Labels ](duplicates.html)
  * [ Categorical data ](categorical.html)
  * [ Nullable integer data type ](integer_na.html)
  * [ Nullable Boolean data type ](boolean.html)
  * [ Chart visualization ](visualization.html)
  * [ Table Visualization ](style.html)
  * [ Group by: split-apply-combine ](groupby.html)
  * [ Windowing operations ](window.html)
  * [ Time series / date functionality ](timeseries.html)
  * [ Time deltas ](timedeltas.html)
  * [ Options and settings ](options.html)
  * [ Enhancing performance ](enhancingperf.html)
  * [ Scaling to large datasets ](scale.html)
  * Sparse data structures 
  * [ Frequently Asked Questions (FAQ) ](gotchas.html)
  * [ Cookbook ](cookbook.html)

  * [ __ ](../index.html)
  * [ User Guide ](index.html)
  * Sparse data... 

#  Sparse data structures  #

pandas provides data structures for efficiently storing sparse data. These are
not necessarily sparse in the typical âmostly 0â. Rather, you can view
these objects as being âcompressedâ where any data matching a specific
value ( ` NaN  ` / missing value, though any value can be chosen, including 0)
is omitted. The compressed values are not actually stored in the array.

    
    
    In [1]: arr = np.random.randn(10)
    
    In [2]: arr[2:-2] = np.nan
    
    In [3]: ts = pd.Series(pd.arrays.SparseArray(arr))
    
    In [4]: ts
    Out[4]: 
    0    0.469112
    1   -0.282863
    2         NaN
    3         NaN
    4         NaN
    5         NaN
    6         NaN
    7         NaN
    8   -0.861849
    9   -2.104569
    dtype: Sparse[float64, nan]
    

Notice the dtype, ` Sparse[float64,  nan]  ` . The ` nan  ` means that
elements in the array that are ` nan  ` arenât actually stored, only the
non- ` nan  ` elements are. Those non- ` nan  ` elements have a ` float64  `
dtype.

The sparse objects exist for memory efficiency reasons. Suppose you had a
large, mostly NA [ ` DataFrame  `
](../reference/api/pandas.DataFrame.html#pandas.DataFrame "pandas.DataFrame")
:

    
    
    In [5]: df = pd.DataFrame(np.random.randn(10000, 4))
    
    In [6]: df.iloc[:9998] = np.nan
    
    In [7]: sdf = df.astype(pd.SparseDtype("float", np.nan))
    
    In [8]: sdf.head()
    Out[8]: 
         0    1    2    3
    0  NaN  NaN  NaN  NaN
    1  NaN  NaN  NaN  NaN
    2  NaN  NaN  NaN  NaN
    3  NaN  NaN  NaN  NaN
    4  NaN  NaN  NaN  N