Package: pandas-intervals
pandas-intervalsPandas extension forDataFrames of intervals (AKAspandas).This library providesivlextension: a standard interface and methods forDataFramesof "interval"-like objects, where each object is specified by astartand anend.The columns"start"and"end"are automatically formatted asfloattypes,The order of the columns is strictly as specified,Interval set operations (intersections, combinations, differences) can be called via theivlproperty on any DataFrame after importing this module.TheIntervalsAccessorobject is also extensible and allows adding columns with default values and specifying how columns are aggregated when combining close intervals.InstallationMinimal requirements:Python >= 3.9Pandas >= 1.5Numpy >= 1.24Install the latest stable version fromPyPi:$ pip install pandas-intervalsOr clone this repo and install:$ git clone https://github.com/BlakeJC94/pandas_intervals
$ cd pandas_intervals
# If Poetry is avaliable:
$ poetry install
# Otherwise Pip requirements can be used:
$ pip install -r requirements.txt
$ pip install .Quick startTo enable this extension, we need to register this extension using the provided setup function.# Top of script/notebook, or top of `__init__.py` in moduleimportpandasaspdfrompandas_intervalsimportsetup_ivl_accessorsetup_ivl_accessor()The core documentation can be accessed within Python:>>>importpandasaspd>>>frompandas_intervalsimportsetup_ivl_accessor>>>setup_ivl_accessor()>>>help(pd.DataFrame.ivl)UsageSay we have two sets of intervalsAandBas specified by:A:     (----]    (----]         (--------------]
  B: (--]   (---]          (----]    (---]    (------] (---]We can represent these sets inpythonusingDataFrames, andimportpandasaspdfrompandas_intervalsimportsetup_ivl_accessorsetup_ivl_accessor()df_a=pd.DataFrame([(50,100),(150,200),(300,450),],).ivl()print(df_a)#    start    end# 0   50.0  100.0# 1  150.0  200.0# 2  300.0  450.0df_b=pd.DataFrame([(10,40),(80,120),(230,280),(330,370),(420,490),(510,550),],).ivl()print(df_b)#    start    end# 0   10.0   40.0# 1   80.0  120.0# 2  230.0  280.0# 3  330.0  370.0# 4  420.0  490.0# 5  510.0  550.0We have all the standard methods available to DataFrames, but we also now have native interval set operations implemented through theivlaccessor:union=df_a.ivl.union(df_b)#     A  :   [----)     [----)#     B  :   [----)             [----)## Result :   [----)     [----)  [----)overlapping=df_a.ivl.overlap()#     A  :   [----)     [----)#               [----)## Result :   [----)#               [----)non_overlapping=df_a.ivl.non_overlap()#     A  :   [----)     [----)#               [----)## Result :              [----)intersection=df_a.ivl.intersection(df_b)#     A  :   [----)        [----)#               [----)#     B  :   [----) [----)        [----)## Result :   [----) [----)#               [----)diff=df_a.ivl.diff(df_b)#     A  :   [----)        [----)#               [----)#     B  :   [----) [----)        [----)## Result :                 [----)symdiff=df_a.ivl.symdiff(df_b)#     A  :   [----)        [----)#               [----)#     B  :   [----) [----)        [----)## Result :                 [----) [----)combined=df_a.ivl.combine()# Optionally accepts intervals DataFrame as arg#     A  :   [----)        [----)#               [----)## Result :   [-------)     [----)complement=df_a.ivl.complement()# Optional kwargs: `left_bound`, `right_bound`#     A  :   [----)        [----)    [---)#               [----)## Result :           [-----)    [----)trunc=df_a.ivl.truncate(df_b)#     A  :   [------------)   [----)     [---)#     B  :      [----)            [----)        [---)## Result :   [--)    [----)   [---)      [---)There's also a few additional methods for filtering and calculating durations and distances on intervals:intervals_between_bounds=df_a.ivl.between(200,400)short_intervals=df_a.ivl.shorter_than(50)long_intervals=df_a.ivl.longest_than(500)padded=df_a.ivl.pad(10)# Optional kwargs: `left_pad`, `right_pad`df_a_min_dist_to_b=df_a.ivl.nearest(df_b)We can also plot intervals using a Plotly backend (if available):df_a.ivl.plot()Multiple groups of intervals can also be plotted on the same graph as well:results=[]fordfin[df_a,df_b,df_c]:df['group']=iresults.append(df)results=pd.concat(results)results.ivl.plot(groupby_cols=['group'])ExtensionsThis interface can easily be extended, we can add additional columns with default values and types.
For example, if we want to create an intervals accessor called"regions"whichHas 2 extra columns ("tag" and "note"),Column "tag" must be specified, but "note" is optional,Column "tag" is an integer, and "note" is a string,Aggregations are done across different values of "tag", and "note" values are combined
into a comma-separated string.We can accomplish this in a relatively small class:importjsonfromosimportPathLikefromtypingimportListimportpandasaspdfrompandas_intervalsimportIntervalsAccessor# Let's define a function to be used when aggregating results across the `notes` columndefcomma_join(x:List[str])->str:return", ".join(sorted({i.strip()forninxifnforiinn.split(",")}))@pd.api.extensions.register_dataframe_accessor("reg")# Name of new accessor, pd.DataFrame.<name>classRegionsAccessor(IntervalsAccessor):# Additional required columns can be specified in a list of tuple# where each tuple is `(column_name, dtype, aggregation)`additional_fields=[("tag","int64","groupby"),# Operations will apply to DFs of tag values separately("note","object",lambdax:",".join(x)),]# Default values for columns can be specified as a dictionary,# columns that don't appear in this list are assumed to be necessarydefault_values={"note":"",}# Add whatever methods/properties you want!@propertydefall_notes(self):returnself.df["note"]defconcat_and_sort(self,df:pd.DataFrame)->pd.DataFrame:df=self.format(df)returnpd.concat([self.df,df],axis=0).sort_values("tag")# Create constructors to parse and validate data from other sources@classmethoddeffrom_json(cls,filepath:PathLike)->pd.DataFrame:withopen(filepath,'r',encoding='utf-8')asf:intervals=json.load(f)assertisinstance(intervals,list)assertlen(intervals)>0assertisinstance(intervals[0],dict)assertall(kinintervals[0]forkin["start","end","tag","note"])df=pd.DataFrame(intervals)returncls.format(df)After defining and running this snippet, we now havepd.DataFrame(..).reg.all_notes()available as a method on any DataFrame,pd.DataFrame(..).reg()will return a formatted DataFrame as specified by the fields inRegionsAccessor.additional_cols,All the methods on theivlaccessor come along for free!ContributingPull requests are most welcome!Dependencies and environments are managed with[poetry](https://python-poetry.org). To get started developing for this package:$ git clone https://github.com/BlakeJC94/pandas_intervals
$ cd pandas_intervals
$ poetry install
$ pytestBenchmarks can be generated and ran with$ pytest --runslow.Branchmainshould be stable, all the latest changes will go onto thedevbranch before being released onmain.Code is tested with[pytest](https://docs.pytest.org)Code is styled using[black](https://black.readthedocs.io)Code is linted with[ruff](https://beta.ruff.rs)Semantic versioningis used in this repoMajor version: rare, substantial changes that break backward compatibilityMinor version: most changes - new features, models or improvementsPatch version: small bug fixes and documentation-only changes