Package: pandas-cub
How to use pandas_cubThe README.ipynb notebook will serve as the documentation and usage guide to pandas_cub.Installationpip install pandas-cubWhat is pandas_cub?pandas_cub is a simple data analysis library that emulates the functionality of the pandas library. The library is not meant for serious work. It was built as an assignment for one of Ted Petrou's Python classes. If you would like to complete the assignment on your own, visitthis repository. There are about 40 steps and 100 tests that you must pass in order to rebuild the library. It is a good challenge and teaches you the fundamentals of how to build your own data analysis library.pandas_cub functionalitypandas_cub has limited functionality but is still capable of a wide variety of data analysis tasks.Subset selection with the bracketsArithmetic and comparison operators (+, -, <, !=, etc...)Aggregation of columns with most of the common functions (min, max, mean, median, etc...)Grouping via pivot tablesString-only methods for columns containing stringsReading in simple comma-separated value filesSeveral other methodspandas_cub DataFramepandas_cub has a single main object, the DataFrame, to hold all of the data. The DataFrame is capable of holding 4 data types - booleans, integers, floats, and strings. All data is stored in NumPy arrays. panda_cub DataFrames have no index (as in pandas). The columns must be strings.Missing value representationBoolean and integer columns will have no missing value representation. The NumPy NaN is used for float columns and the Python None is used for string columns.Code Examplespandas_cub syntax is very similar to pandas, but implements much fewer methods. The below examples will cover just about all of the API.Reading data withread_csvpandas_cub consists of a single function,read_csv, that has a single parameter, the location of the file you would like to read in as a DataFrame. This function can only handle simple CSV's and the delimiter must be a comma. A sample employee dataset is provided in the data directory. Notice that the visual output of the DataFrame is nearly identical to that of a pandas DataFrame. Theheadmethod returns the first 5 rows by default.importpandas_cubaspdcdf=pdc.read_csv('data/employee.csv')df.head()deptracegendersalary0Houston Police Department-HPDWhiteMale452791Houston Fire Department (HFD)WhiteMale631662Houston Police Department-HPDBlackMale666143Public Works & Engineering-PWEAsianMale716804Houston Airport System (HAS)WhiteMale42390DataFrame propertiesTheshapeproperty returns a tuple of the number of rows and columnsdf.shape(1535, 4)Thelenfunction returns just the number of rows.len(df)1535Thedtypesproperty returns a DataFrame of the column names and their respective data type.df.dtypesColumn NameData Type0deptstring1racestring2genderstring3salaryintThecolumnsproperty returns a list of the columns.df.columns['dept', 'race', 'gender', 'salary']Set new columns by assigning thecolumnsproperty to a list.df.columns=['department','race','gender','salary']df.head()departmentracegendersalary0Houston Police Department-HPDWhiteMale452791Houston Fire Department (HFD)WhiteMale631662Houston Police Department-HPDBlackMale666143Public Works & Engineering-PWEAsianMale716804Houston Airport System (HAS)WhiteMale42390Thevaluesproperty returns a single numpy array of all the data.df.valuesarray([['Houston Police Department-HPD', 'White', 'Male', 45279],
       ['Houston Fire Department (HFD)', 'White', 'Male', 63166],
       ['Houston Police Department-HPD', 'Black', 'Male', 66614],
       ...,
       ['Houston Police Department-HPD', 'White', 'Male', 43443],
       ['Houston Police Department-HPD', 'Asian', 'Male', 55461],
       ['Houston Fire Department (HFD)', 'Hispanic', 'Male', 51194]],
      dtype=object)Subset selectionSubset selection is handled with the brackets. To select a single column, place that column name in the brackets.df['race'].head()race0White1White2Black3Asian4WhiteSelect multiple columns with a list of strings.df[['race','salary']].head()racesalary0White452791White631662Black666143Asian716804White42390Simultaneously select rows and columns by passing the brackets the row selection followed by the column selection separated by a comma. Here we use integers for rows and strings for columns.rows=[10,50,100]cols=['salary','race']df[rows,cols]salaryrace077076Black181239White281239WhiteYou can use integers for the columns as well.rows=[10,50,100]cols=[2,0]df[rows,cols]genderdepartment0MaleHouston Police Department-HPD1MaleHouston Police Department-HPD2MaleHouston Police Department-HPDYou can use a single integer and not just a list.df[99,3]salary066614Or a single string for the columnsdf[99,'salary']salary066614You can use a slice for the rowsdf[20:100:10,['race','gender']]racegender0WhiteMale1WhiteMale2HispanicMale3WhiteMale4WhiteMale5HispanicMale6HispanicMale7BlackFemaleYou can also slice the columns with either integers or stringsdf[20:100:10,:2]departmentrace0Houston Police Department-HPDWhite1Houston Fire Department (HFD)White2Houston Police Department-HPDHispanic3Houston Police Department-HPDWhite4Houston Fire Department (HFD)White5Houston Police Department-HPDHispanic6Houston Fire Department (HFD)Hispanic7Houston Police Department-HPDBlackdf[20:100:10,'department':'gender']departmentracegender0Houston Police Department-HPDWhiteMale1Houston Fire Department (HFD)WhiteMale2Houston Police Department-HPDHispanicMale3Houston Police Department-HPDWhiteMale4Houston Fire Department (HFD)WhiteMale5Houston Police Department-HPDHispanicMale6Houston Fire Department (HFD)HispanicMale7Houston Police Department-HPDBlackFemaleYou can do boolean selection if you pass the brackets a one-column boolean DataFrame.filt=df['salary']>100000filt.head()salary0False1False2False3False4Falsedf[filt].head()departmentracegendersalary0Public Works & Engineering-PWEWhiteMale1079621Health & Human ServicesBlackMale1804162Houston Fire Department (HFD)HispanicMale1652163Health & Human ServicesWhiteFemale1007914Houston Airport System (HAS)WhiteMale120916df[filt,['race','salary']].head()racesalary0White1079621Black1804162Hispanic1652163White1007914White120916Assigning ColumnsYou can only assign an entire new column or overwrite an old one. You cannot assign a subset of the data. You can assign a new column with a single value like this:df['bonus']=1000df.head()departmentracegendersalarybonus0Houston Police Department-HPDWhiteMale4527910001Houston Fire Department (HFD)WhiteMale6316610002Houston Police Department-HPDBlackMale6661410003Public Works & Engineering-PWEAsianMale7168010004Houston Airport System (HAS)WhiteMale423901000You can assign with a numpy array the same length as a column.importnumpyasnpdf['bonus']=np.random.randint(100,5000,len(df))df.head()departmentracegendersalarybonus0Houston Police Department-HPDWhiteMale4527935361Houston Fire Department (HFD)WhiteMale6316612962Houston Police Department-HPDBlackMale666145113Public Works & Engineering-PWEAsianMale7168042674Houston Airport System (HAS)WhiteMale423903766You can assign a new column with a one column DataFrame.df['salary']+df['bonus']salary04881516446226712537594744615651100016537387185348832575957918......1525329361526492941527342181528827951529104900153046408153167050153247368153360013153452624df['total salary']=df['salary']+df['bonus']df.head()departmentracegendersalarybonustotal salary0Houston Police Department-HPDWhiteMale452793536488151Houston Fire Department (HFD)WhiteMale631661296644622Houston Police Department-HPDBlackMale66614511671253Public Works & Engineering-PWEAsianMale716804267759474Houston Airport System (HAS)WhiteMale42390376646156Arithmetic and comparison operatorsdf1=df[['salary','bonus']]*5df1.head()salarybonus0226395176801315830648023330702555335840021335421195018830df1=df[['salary','bonus']]>100000df1.head()salarybonus0FalseFalse1FalseFalse2FalseFalse3FalseFalse4FalseFalsedf1=df['race']=='White'df1.head()race0True1True2False3False4TrueAggregationMost of the common aggregation methods are available. They only work down the columns and not across the rows.df.min()departmentracegendersalarybonustotal salary0Health & Human ServicesAsianFemale2496010125913Columns that the aggregation does not work are dropped.df.mean()salarybonustotal salary056278.7462594.28358873.029df.argmax()departmentracegendersalarybonustotal salary03001451516145df['salary'].argmin()salary0347Check if all salaries are greater than 20000df1=df['salary']>20000df1.all()salary0TrueCount the number of non-missing valuesdf.count()departmentracegendersalarybonustotal salary0153515351535153515351535Get number of unique values.df.nunique()departmentracegendersalarybonustotal salary065254813181524Non-Aggregating MethodsThese are methods that do not return a single value.Get the unique values of each column. Theuniquemethod returns a list of DataFrames containing the unique values for each column.dfs=df.unique()dfs[0]department0Health & Human Services1Houston Airport System (HAS)2Houston Fire Department (HFD)3Houston Police Department-HPD4Parks & Recreation5Public Works & Engineering-PWEdfs[1]race0Asian1Black2Hispanic3Native American4Whitedfs[2]gender0Female1MaleRename columns with a dictionary.df.rename({'department':'dept','bonus':'BONUS'}).head()deptracegendersalaryBONUStotal salary0Houston Police Department-HPDWhiteMale452793536488151Houston Fire Department (HFD)WhiteMale631661296644622Houston Police Department-HPDBlackMale66614511671253Public Works & Engineering-PWEAsianMale716804267759474Houston Airport System (HAS)WhiteMale42390376646156Drop columns with a string or list of strings.df.drop('race').head()departmentgendersalarybonustotal salary0Houston Police Department-HPDMale452793536488151Houston Fire Department (HFD)Male631661296644622Houston Police Department-HPDMale66614511671253Public Works & Engineering-PWEMale716804267759474Houston Airport System (HAS)Male42390376646156df.drop(['race','gender']).head()departmentsalarybonustotal salary0Houston Police Department-HPD452793536488151Houston Fire Department (HFD)631661296644622Houston Police Department-HPD66614511671253Public Works & Engineering-PWE716804267759474Houston Airport System (HAS)42390376646156Non-aggregating methods that keep all columnsThe next several methods are non-aggregating methods that return a DataFrame with the same exact shape as the original. They only work on boolean, integer and float columns and ignore string columns.Absolute valuedf.abs().head()departmentracegendersalarybonustotal salary0Houston Police Department-HPDWhiteMale452793536488151Houston Fire Department (HFD)WhiteMale631661296644622Houston Police Department-HPDBlackMale66614511671253Public Works & Engineering-PWEAsianMale716804267759474Houston Airport System (HAS)WhiteMale42390376646156Cumulative min, max, and sumdf.cummax().head()departmentracegendersalarybonustotal salary0Houston Police Department-HPDWhiteMale452793536488151Houston Fire Department (HFD)WhiteMale631663536644622Houston Police Department-HPDBlackMale666143536671253Public Works & Engineering-PWEAsianMale716804267759474Houston Airport System (HAS)WhiteMale71680426775947Clip values to be within a range.df.clip(40000,60000).head()departmentracegendersalarybonustotal salary0Houston Police Department-HPDWhiteMale4527940000488151Houston Fire Department (HFD)WhiteMale6000040000600002Houston Police Department-HPDBlackMale6000040000600003Public Works & Engineering-PWEAsianMale6000040000600004Houston Airport System (HAS)WhiteMale423904000046156Round numeric columnsdf.round(-3).head()departmentracegendersalarybonustotal salary0Houston Police Department-HPDWhiteMale450004000490001Houston Fire Department (HFD)WhiteMale630001000640002Houston Police Department-HPDBlackMale670001000670003Public Works & Engineering-PWEAsianMale720004000760004Houston Airport System (HAS)WhiteMale42000400046000Copy the DataFramedf.copy().head()departmentracegendersalarybonustotal salary0Houston Police Department-HPDWhiteMale452793536488151Houston Fire Department (HFD)WhiteMale631661296644622Houston Police Department-HPDBlackMale66614511671253Public Works & Engineering-PWEAsianMale716804267759474Houston Airport System (HAS)WhiteMale42390376646156Take the nth difference.df.diff(2).head(10)departmentracegendersalarybonustotal salary0Houston Police Department-HPDWhiteMalenannannan1Houston Fire Department (HFD)WhiteMalenannannan2Houston Police Department-HPDBlackMale21335.000-3025.00018310.0003Public Works & Engineering-PWEAsianMale8514.0002971.00011485.0004Houston Airport System (HAS)WhiteMale-24224.0003255.000-20969.0005Public Works & Engineering-PWEWhiteMale36282.000-2228.00034054.0006Houston Fire Department (HFD)HispanicMale10254.000-2672.0007582.0007Health & Human ServicesBlackMale72454.0002893.00075347.0008Public Works & Engineering-PWEBlackMale-22297.0001134.000-21163.0009Health & Human ServicesBlackMale-125147.000-2283.000-127430.000Find the nth percentage change.df.pct_change(2).head(10)departmentracegendersalarybonustotal salary0Houston Police Department-HPDWhiteMalenannannan1Houston Fire Department (HFD)WhiteMalenannannan2Houston Police Department-HPDBlackMale0.471-0.8550.3753Public Works & Engineering-PWEAsianMale0.1352.2920.1784Houston Airport System (HAS)WhiteMale-0.3646.370-0.3125Public Works & Engineering-PWEWhiteMale0.506-0.5220.4486Houston Fire Department (HFD)HispanicMale0.242-0.7100.1647Health & Human ServicesBlackMale0.6711.4190.6858Public Works & Engineering-PWEBlackMale-0.4241.037-0.3949Health & Human ServicesBlackMale-0.694-0.463-0.688Sort the DataFrame by one or more columnsdf.sort_values('salary').head()departmentracegendersalarybonustotal salary0Houston Police Department-HPDBlackFemale24960953259131Public Works & Engineering-PWEHispanicMale261044258303622Public Works & Engineering-PWEBlackFemale261253247293723Houston Airport System (HAS)HispanicFemale26125832269574Houston Airport System (HAS)BlackFemale26125246128586Sort descendingdf.sort_values('salary',asc=False).head()departmentracegendersalarybonustotal salary0Houston Fire Department (HFD)WhiteMale21058837242143121Houston Police Department-HPDWhiteMale1995968482004442Houston Airport System (HAS)BlackMale18619217781879703Health & Human ServicesBlackMale18041649321853484Public Works & Engineering-PWEWhiteFemale1783312124180455Sort by multiple columnsdf.sort_values(['race','salary']).head()departmentracegendersalarybonustotal salary0Houston Airport System (HAS)AsianFemale261254446305711Houston Police Department-HPDAsianMale279142855307692Houston Police Department-HPDAsianMale281692572307413Public Works & Engineering-PWEAsianMale289952874318694Public Works & Engineering-PWEAsianMale30347493835285Randomly sample the DataFramedf.sample(n=3)departmentracegendersalarybonustotal salary0Houston Fire Department (HFD)WhiteMale625402995655351Public Works & Engineering-PWEWhiteMale633361547648832Houston Police Department-HPDWhiteMale52514115053664Randomly sample a fractiondf.sample(frac=.005)departmentracegendersalarybonustotal salary0Houston Police Department-HPDHispanicFemale603471200615471Public Works & Engineering-PWEBlackMale491093598527072Health & Human ServicesBlackFemale489844602535863Houston Police Department-HPDWhiteMale554612813582744Houston Airport System (HAS)BlackFemale292861877311635Houston Police Department-HPDAsianMale666144480710946Houston Fire Department (HFD)WhiteMale28024447532499Sample with replacementdf.sample(n=10000,replace=True).head()departmentracegendersalarybonustotal salary0Parks & RecreationBlackFemale310751665327401Public Works & Engineering-PWEHispanicMale67038644676822Houston Police Department-HPDBlackMale370241532385563Health & Human ServicesBlackFemale574333106605394Public Works & Engineering-PWEBlackMale5337392454297String-only methodsUse thestraccessor to call methods available just to string columns. Pass the name of the string column as the first parameter for all these methods.df.str.count('department','P').head()department0210223240df.str.lower('department').head()department0houston police department-hpd1houston fire department (hfd)2houston police department-hpd3public works & engineering-pwe4houston airport system (has)df.str.find('department','Houston').head()department0010203-140Groupingpandas_cub provides thevalue_countsmethod for simple frequency counting of unique values andpivot_tablefor grouping and aggregating.Thevalue_countsmethod returns a list of DataFrames, one for each column.dfs=df[['department','race','gender']].value_counts()dfs[0]departmentcount0Houston Police Department-HPD5701Houston Fire Department (HFD)3652Public Works & Engineering-PWE3413Health & Human Services1034Houston Airport System (HAS)1035Parks & Recreation53dfs[1]racecount0White5421Black5182Hispanic3813Asian874Native American7dfs[2]gendercount0Male11351Female400If your DataFrame has one column, a DataFrame and not a list is returned. You can also return the relative frequency by setting thenormalizeparameter toTrue.df['race'].value_counts(normalize=True)racecount0White0.3531Black0.3372Hispanic0.2483Asian0.0574Native American0.005Thepivot_tablemethod allows to group by one or two columns and aggregate values from another column. Let's find the average salary for each race and gender. All parameters must be strings.df.pivot_table(rows='race',columns='gender',values='salary',aggfunc='mean')raceFemaleMale0Asian58304.22260622.9571Black48133.38251853.0002Hispanic44216.96055493.0643Native American58844.33368850.5004White66415.52863439.196If you don't providevaluesoraggfuncthen by default it will return frequency (a contingency table).df.pivot_table(rows='race',columns='gender')raceFemaleMale0Asian18691Black2073112Hispanic1002813Native American344White72470You can group by just a single column.df.pivot_table(rows='department',values='salary',aggfunc='mean')departmentmean0Health & Human Services51324.9811Houston Airport System (HAS)53990.3692Houston Fire Department (HFD)59960.4413Houston Police Department-HPD60428.7464Parks & Recreation39426.1515Public Works & Engineering-PWE50207.806df.pivot_table(columns='department',values='salary',aggfunc='mean')Health & Human ServicesHouston Airport System (HAS)Houston Fire Department (HFD)Houston Police Department-HPDParks & RecreationPublic Works & Engineering-PWE051324.98153990.36959960.44160428.74639426.15150207.806