Package: sidetable
sidetablesidetable started as a supercharged combination of pandasvalue_countspluscrosstabthat builds simple but useful summary tables of your pandas DataFrame. It has since expanded
to provide support for common and useful pandas tasks such as adding subtotals to your
DataFrame or flattening hierarchical columns.Usage is straightforward. Install andimport sidetable. Then access it through the
new.stbaccessor on your DataFrame.For the Titanic data:df.stb.freq(['class'])will build a frequency table like this:classcountpercentcumulative_countcumulative_percent0Third49155.106649155.10661First21624.242470779.3492Second18420.651891100You can also summarize missing values withdf.stb.missing():missingtotalpercentdeck68889177.2166age17789119.8653embarked28910.224467embark_town28910.224467survived08910pclass08910sex08910sibsp08910parch08910fare08910class08910who08910adult_male08910alive08910alone08910You can group the data and add subtotals and grand totals withstb.subtotal():df.groupby(['sex','class']).agg({'fare':['sum']}).stb.subtotal()faresumsexclassfemaleFirst9975.8250Second1669.7292Third2321.1086female - subtotal13966.6628maleFirst8201.5875Second2132.1125Third4393.5865male - subtotal14727.2865grand_total28693.9493You can also turn a hierarchical column structure into this:titanic.groupby(['embark_town','class','sex']).agg({'fare':['sum'],'age':['mean']}).unstack().stb.flatten()embark_townclassfare_sum_femalefare_sum_maleage_mean_femaleage_mean_male0CherbourgFirst4972.533928.5436.052640.11111CherbourgSecond176.879254.21219.142925.93752CherbourgThird337.983402.14614.062525.01683QueenstownFirst909033444QueenstownSecond24.712.3530575QueenstownThird340.159465.04622.8528.14296SouthamptonFirst4753.294183.0532.704541.89727SouthamptonSecond1468.151865.5529.719730.87598SouthamptonThird1642.973526.3923.223726.5748sidetable has several useful features:See total counts and their relative percentages in one table. This is roughly equivalent to combining the
output ofvalue_counts()andvalue_counts(normalize=True)into one table.Include cumulative totals and percentages to better understand your thresholds.
ThePareto principleapplies to many different scenarios
and this function makes it easy to see how your data is cumulatively distributed.Aggregate multiple columns together to see frequency counts for grouped data.Provide a threshold point above which all data is grouped into a single bucket. This is useful for
quickly identifying the areas to focus your analysis.Get a count of the missing values in your data.Count the number of unique values for each column.Add grand totals on any DataFrame and subtotals to any grouped DataFrame.Pretty print columnsTable of Contents:Quick StartRationaleInstallationUsagefreqcountsmissingsubtotalflattenprettyprintCaveatsTODOContributingCreditsQuickstartFor the impatient:$ python -m pip install sidetableimportsidetableimportpandasaspd# Create your DataFramedf=pd.read_csv(myfile.csv)# Build a frequency table for one or more columnsdf.stb.freq(['column1','column2'])# See what data is missingdf.stb.missing()# Group data and add a subtotaldf.groupby(['column1','column2'])['col3'].sum().stb.subtotal()That's it.Read on for more details and more examples of what you can do sidetable.RationaleThe idea behind sidetable is that there are a handful of useful data analysis tasks that
you might run on any data set early in the data analysis process. While each of these
tasks can be done in a handful of lines of pandas code, it is a lot of typing and
difficult to remember.In addition to providing useful functionality, this project is also a test to see how to
build custom accessors using some of pandas relatively new API. I am hopeful this can
serve as a model for other projects whether open source or just for your own usage.
Please check out therelease announcementfor more
information about the usage and how to use this as a model for your own projects.The solutions in sidetable are heavily based on three sources:Thistweet threadby Peter BaumgartnerAnexcellent articleby Steve Miller that lays out many of the code concepts incorporated into sidetable.Ted Petrou'sposton finding the percentage of missing values in a DataFrame.I very much appreciate the work that all three authors did to point me in this direction.Installation$  python -m pip install -U sidetableThis is the preferred method to install sidetable, as it will always
install the most recent stable release. sidetable requires pandas 1.0 or higher and no
additional dependencies. It should run anywhere that pandas runs.If you prefer to use conda, sidetable is available on conda-forge:$ conda install -c conda-forge sidetableUsageimportpandasaspdimportsidetableimportseabornassnsdf=sns.load_dataset('titanic')sidetable uses the pandas DataFrameaccessor apito add a.stbaccessor to all of your DataFrames. Once youimport sidetableyou are ready to
go. In these examples, I will be using seaborn's Titanic dataset as an example but
seaborn is not a direct dependency.freqIf you have usedvalue_counts()before, you have probably wished it were easier to
combine the values with percentage distribution.df['class'].value_counts()Third491First216Second184Name:class,dtype:int64df['class'].value_counts(normalize=True)Third0.551066First0.242424Second0.206510Name:class,dtype:float64Which can be done, but is messy and a lot of typing and remembering:pd.concat([df['class'].value_counts().rename('count'),df['class'].value_counts(normalize=True).mul(100).rename('percentage')],axis=1)countpercentageThird49155.1066First21624.2424Second18420.651Using sidetable is much simpler and you get cumulative totals, percents and more flexibility:df.stb.freq(['class'])classcountpercentcumulative_countcumulative_percent0Third49155.106649155.10661First21624.242470779.3492Second18420.651891100If you want to style the results so percentages and large numbers are easier to read,
usestyle=True:df.stb.freq(['class'],style=True)classcountpercentcumulative_countcumulative_percent0Third49155.11%49155.11%1First21624.24%70779.35%2Second18420.65%891100.00%In addition, you can group columns together. If we want to see the breakdown among
class and sex:df.stb.freq(['sex','class'])sexclasscountpercentcumulative_countcumulative_percent0maleThird34738.94534738.9451femaleThird14416.161649155.10662maleFirst12213.692561368.79913maleSecond10812.121272180.92034femaleFirst9410.549981591.47035femaleSecond768.52974891100You can use as many groupings as you would like.By default, sidetable counts the data. However, you can specify avalueargument to
indicate that the data should be summed based on the data in another column.
For this data set, we can see how the fares are distributed by class:df.stb.freq(['class'],value='fare')classfarepercentcumulative_farecumulative_percent0First18177.463.349318177.463.34931Third6714.723.401124892.186.75042Second3801.8413.249628693.9100Another feature of sidetable is that you can specify a threshold. For many data analysis,
you may want to break down into large groupings to focus on and ignore others. You can use
thethreshargument to define a threshold and group all entries above that threshold
into an "other" grouping:df.stb.freq(['class','who'],value='fare',thresh=80)classwhofarepercentcumulative_farecumulative_percent0Firstwoman9492.9433.08349492.9433.08341Firstman7848.1827.351317341.160.43482Thirdman3617.5312.607320958.673.0423Secondman1886.366.574062284579.61614othersothers5848.9520.383928693.9100You can further customize by specifying the label to use for all the others:df.stb.freq(['class','who'],value='fare',thresh=80,other_label='All others')classwhofarepercentcumulative_farecumulative_percent0Firstwoman9492.9433.08349492.9433.08341Firstman7848.1827.351317341.160.43482Thirdman3617.5312.607320958.673.0423Secondman1886.366.574062284579.61614All othersAll others5848.9520.383928693.9100countsThecounts()function shows how many unique values are in each column as well as
the most and least frequent values & their total counts. This summary view can help you determine if you need
to convert data to a categorical value. It can also help you understand the high
level structure of your data.df.stb.counts()countuniquemost_freqmost_freq_countleast_freqleast_freq_countsurvived891205491342sex8912male577female314adult_male8912True537False354alive8912no549yes342alone8912True537False354pclass891334912184embarked8893S644Q77class8913Third491Second184who8913man537child83embark_town8893Southampton644Queenstown77sibsp8917060855parch8917067861deck2037C59G4age7148824.03020.51fare8912488.054363.35831By default, all data types are included but you may use theexcludeandincludeparameters
to select specific types of columns. The syntax is the same as pandasselect_dtypesFor example,df.stb.counts(exclude='number')countuniquemost_freqmost_freq_countleast_freqleast_freq_countsex8912male577female314adult_male8912True537False354alive8912no549yes342alone8912True537False354embarked8893S644Q77class8913Third491Second184who8913man537child83embark_town8893Southampton644Queenstown77deck2037C59G4missingsidetable also includes a summary table that shows the missing values in
your data by count and percentage of total missing values in a column.df.stb.missing()missingtotalpercentdeck68889177.2166age17789119.8653embarked28910.224467embark_town28910.224467survived08910pclass08910sex08910sibsp08910parch08910fare08910class08910who08910adult_male08910alive08910alone08910If you wish to see the results with styles applied to the Percent and Total column,
use:df.stb.missing(style=True)missingtotalpercentdeck68889177.22%age17789119.87%embarked28910.22%embark_town28910.22%survived08910pclass08910sex08910sibsp08910parch08910fare08910class08910who08910adult_male08910alive08910alone08910Finally, you can exclude the columns that have 0 missing values using
theclip_0=Trueparameter:df.stb.missing(clip_0=True,style=True)missingtotalpercentdeck68889177.22%age17789119.87%embarked28910.22%embark_town28910.22%subtotalAnother useful function is the subtotal function. Trying to add a subtotal
to grouped pandas data is not easy. sidetable adds asubtotal()function that
makes adds a subtotal at one or more levels of a DataFrame.The subtotal function can be applied to a simple DataFrame in order to add a Grand Total
label:df.stb.subtotal()survivedpclasssexagesibspparchfareembarkedclasswhoadult_maledeckembark_townalivealone88711female190030SFirstwoman0BSouthamptonyes188803femalenan1223.45SThirdwoman0nanSouthamptonno088911male260030CFirstman1CCherbourgyes189003male32007.75QThirdman1nanQueenstownno1grand_total3422057nan21205.246634028693.9nannannan537nannannan537The real power of subtotal is being able to add it to one or more levels of your
grouped data. For example, you can group the data and add a subtotal at each level:df.groupby(['sex','class','embark_town']).agg({'fare':['sum']}).stb.subtotal()Which yields this view (truncated for simplicity):faresumsexclassembark_townfemaleFirstCherbourg4972.5333Queenstown90.0000Southampton4753.2917female | First - subtotal9815.8250SecondCherbourg176.8792Queenstown24.7000Southampton1468.1500female | Second - subtotal1669.7292ThirdCherbourg337.9833Queenstown340.1585Southampton1642.9668female | Third - subtotal2321.1086female - subtotal13806.6628maleFirstCherbourg3928.5417Queenstown90.0000By default, every level in the DataFrame will be subtotaled but you can control this behavior
by using thesub_levelargument. For instance, you can subtotal onsexandclassby
passing the argumentsub_level=[1,2]summary_table=df.groupby(['sex','class','embark_town']).agg({'fare':['sum']})summary_table.stb.subtotal(sub_level=[1,2])Thesubtotalfunction also allows the user to configure the labels and separators used in
the subtotal and Grand Total by using thegrand_label,sub_label,show_sepandseparguments.flattenWhen grouping and pivoting data, you can end up with a DataFrame that has a multiindex.
Often times, you want a simple flat representation of the data.For example, we can build a table using agroupby()plusunstack()that looks like this:df.groupby(['embark_town','class','sex']).agg({'fare':['sum'],'age':['mean']}).unstack()fareagesummeansexfemalemalefemalemaleembark_townclassCherbourgFirst4972.53333928.541736.05263240.111111Second176.8792254.212519.14285725.937500Third337.9833402.146214.06250025.016800QueenstownFirst90.000090.000033.00000044.000000Second24.700012.350030.00000057.000000Third340.1585465.045822.85000028.142857SouthamptonFirst4753.29174183.045832.70454541.897188Second1468.15001865.550029.71969730.875889Third1642.96683526.394523.22368426.574766If you wish to flatten it, usestb.flatten():df.groupby(['embark_town','class','sex']).agg({'fare':['sum'],'age':['mean']}).unstack().stb.flatten()embark_townclassfare_sum_femalefare_sum_maleage_mean_femaleage_mean_male0CherbourgFirst4972.533928.5436.052640.11111CherbourgSecond176.879254.21219.142925.93752CherbourgThird337.983402.14614.062525.01683QueenstownFirst909033444QueenstownSecond24.712.3530575QueenstownThird340.159465.04622.8528.14296SouthamptonFirst4753.294183.0532.704541.89727SouthamptonSecond1468.151865.5529.719730.87598SouthamptonThird1642.973526.3923.223726.5748flatten will also take additional arguments:Add a custom separator using thesepargument -stb.flatten(sep='|')Control whether or not to reset the index usingresetargument -stb.flatten(reset=False)Reorganize the output levels usinglevelsargumentlevels=2levelscan also take a list of valid levels if you want to reorganize the displaylevels=[0,2]fares=df.groupby(['embark_town','class','sex']).agg({'fare':['sum'],'age':['mean']}).unstack()fares.stb.flatten(sep='|',reset=False,levels=[0,2])fare|femalefare|malefare|femalefare|maleage|femaleage|maleembark_townclassCherbourgFirst4972.53333928.5417115.64030993.53670736.05263240.111111Second176.8792254.212525.26845725.42125019.14285725.937500Third337.9833402.146214.6949269.35223714.06250025.016800QueenstownFirst90.000090.000090.00000090.00000033.00000044.000000Second24.700012.350012.35000012.35000030.00000057.000000Third340.1585465.045810.30783311.92425122.85000028.142857SouthamptonFirst4753.29174183.045899.02691052.94994732.70454541.897188Second1468.15001865.550021.91268719.23247429.71969730.875889Third1642.96683526.394518.67007713.30714923.22368426.574766prettyprintThis function interprets the magnitude of your numeric results and returns a nicely
formatted version of all the numbers. This can be used on a full DataFrame or during
your analysis of aggregated data.For instance, if you are summarizing data, you may get something that looks like this:df.groupby(['pclass','sex']).agg({'fare':'sum'})farepclasssex1female9975.8250male8201.58752female1669.7292male2132.11253female2321.1086male4393.5865Usestb.pretty()to format it nicely so you can have the same order or magnitude for all numbers:df.groupby(['pclass','sex']).agg({'fare':'sum'}).div(df['fare'].sum()).stb.pretty()farepclasssex1female9.98kmale8.20k2female1.67kmale2.13k3female2.32kmale4.39kHere's an example of a percentage format:df.groupby(['pclass','sex']).agg({'fare':'sum'}).div(df['fare'].sum()).stb.pretty(precision=0,caption="Fare Percentage")Fare Percentagefarepclasssex1female35%male29%2female6%male7%3female8%male15%Behind the scenes,prettywill attempt to normalize the values. You can control theprecision,rowsadd acaption.Caveatssidetable supports grouping on any data type in a pandas DataFrame. This means that
you could try something like:df.stb.freq(['fare'])In some cases where there are a fairly small discrete number of this may be useful. However,
if you have a lot of unique values, you shouldbin the datafirst. In the example, above the data would include 248 rows and not be terribly useful.One alternative could be:df['fare_bin']=pd.qcut(df['fare'],q=4,labels=['low','medium','high','x-high'])df.stb.freq(['fare_bin'])fare_bincountpercentcumulative_countcumulative_percent0medium22425.140322425.14031low22325.028144750.16842x-high22224.915866975.08423high22224.9158891100The other caveat is that null or missing values can cause data to drop out while aggregating.
For instance, if we look at thedeckvariable, there are a lot of missing values.df.stb.freq(['deck'])deckcountpercentcumulative_countcumulative_percent0C5929.0645929.0641B4723.152710652.21672D3316.256213968.47293E3215.763517184.23654A157.3891618691.62565F136.4039419998.02966G41.97044203100The total cumulative count only goes up to 203 not the 891 we have seen in other examples.
Future versions of sidetable may handle this differently. For now, it is up to you to
decide how best to handle unknowns. For example, this version of the Titanic data set
has a categorical value fordeckso usingfillnarequires an extra step:df['deck_fillna']=df['deck'].cat.add_categories('UNK').fillna('UNK')df.stb.freq(['deck_fillna'])deck_fillnacountpercentcumulative_countcumulative_percent0UNK68877.216668877.21661C596.6217774783.83842B475.2749779489.11343D333.703782792.81714E323.5914785996.40855A151.683587498.0926F131.4590388799.55117G40.448934891100Another variant is that there might be certain groupings where there are no valid counts.For instance, if we look at thedeckandclass:df.stb.freq(['deck','class'])deckclasscountpercentcumulative_countcumulative_percent0CFirst5929.0645929.0641BFirst4723.152710652.21672DFirst2914.285713566.50253EFirst2512.315316078.81774AFirst157.3891617586.20695FSecond83.9408918390.14786FThird52.4630518892.61087GThird41.9704419294.58138ESecond41.9704419696.55179DSecond41.9704420098.522210EThird31.47783203100There are only 11 combinations. If we want to see all - even if there are not any passengers
fitting that criteria, useclip_0=Falsedf.stb.freq(['deck','class'],clip_0=False)deckclasscountpercentcumulative_countcumulative_percent0CFirst5929.0645929.0641BFirst4723.152710652.21672DFirst2914.285713566.50253EFirst2512.315316078.81774AFirst157.3891617586.20695FSecond83.9408918390.14786FThird52.4630518892.61087GThird41.9704419294.58138ESecond41.9704419696.55179DSecond41.9704420098.522210EThird31.4778320310011GSecond0020310012GFirst0020310013FFirst0020310014DThird0020310015CThird0020310016CSecond0020310017BThird0020310018BSecond0020310019AThird0020310020ASecond00203100In many cases this might be too much data, but sometimes the fact that a combination is
missing could be insightful.The final caveat relates tosubtotal. When working with thesubtotalfunction, sidetable
convert a Categorical MultiIndex to a plain index in order to easily add the subtotal labels.TODOHandle NaN values more effectivelyOffer binning options for continuous variablesOffer more options, maybe plotting?ContributingContributions are welcome, and they are greatly appreciated! Every
little bit helps, and credit will always be given. If you have a new idea for a simple table
that we should add, please submit a ticket.For more info please clickhereCreditsThis package was created with Cookiecutter and theoldani/cookiecutter-simple-pypackageproject template.
The code used in this package is heavily based on the posts from Peter Baumgartner, Steve Miller
and Ted Petrou. Thank you!Cookiecutteroldani/cookiecutter-simple-pypackagePeter Baumgartner -tweet threadSteve Miller -articleTed Petrou -post