pydata/pandas — Issue #817: Groupby performance discrepency in timestamped data
Author: adamklein
Created: 2012-02-23 15:54:34+00:00
State: open
Labels: Enhancement

From mailing list

serie = pandas.io.parsers.read_csv(f, parse_dates=True,
date_parser=dateParser, index_col=0)
dateRange = pandas.DateRange(start, end, offset=5 *
pandas.datetools.Minute())
grouped = serie.groupby(dateRange.asof)

# version 1
t = time()
for date in serie.index:
k = grouped.grouper(date) # returns the key of the group where
date belongs
g = grouped.get_group(k)
print time()-t

# version 2
t = time()
for date in serie.index:
k = grouped.grouper(date)
g = serie.ix[grouped.groups[k]]
print time()-t

serie is something looking like this (financial data indexed with
datetime)

Index: 476640 entries, 2011-01-03 00:00:00 to 2011-11-29 23:59:00
Data columns:
Open 476640 non-null values
High 476640 non-null values
Low 476640 non-null values
Close 476640 non-null values
Volume 476640 non-null values
dtypes: float64(5)

For 100.000 elts, version 1 performs in 480 secs, while version 2
takes only 25 secs
For the full 460.000 elts, we then get about 40 mins and 135 secs