The 9 must know functions to handle any pandas dataframe | by Alejandra Vlerick | Towards Data Science

Five functions to get to know your dataset
Filter your dataframe by value
Had datetime type
Merge multiple datasets
Handle columns
Group by value
Apply a function
Use the lambda method
If you’re not already a member, get your Medium Membership here!
The first thing to do after loading a dataset is to take a good look at the information in your dataframe. For this the following functions give you exactly what you need.
Head() prints the first 5 rows of you dataset including column header and the content of each row.
df.head()
Info() prints the column header and the data type stored in each column. This function is extremely useful when you are trying to understand what values need to change types in order to apply functions to them. Integers that are stored as string will not be added together until you transform them into integers.
df.info()
Shape() gives the size of the dataframe in the format (‘row’ x ‘column’).
df.shape
Describe() gives the mean, median, standard deviation and percentiles of all the numerical values in your dataset. I would suggest using describe after making sure all the numerical values are stored as integers or floats.
df.describe()
Unique() allows you to look at the unique values in each column. This is particularly useful when you have categorical data and you want to understand the categories.
df['columnName'].unique()
Find the most repeated values in a dataset for a given column.
df.column.value_counts().head(20)
Filtering is an essential skill that is required when dealing with any pandas dataframe. To compare the values in a columns with the filter, operators are used.
The filtering can be done for numerical values, but also for qualitative or categorical values.
df = df[df.value > 0 ]df = df[df.column != 'value']
Various filters can be applied by using bitwise operators. In the examples below the OR and AND operators are used to create two new dataframes which are filtered down versions of the original dataframe.
df1 = df[(df.Party == ‘Conservative’) | (df.Party == ‘Labour’)]df2 = df[(df.Party == ‘Conservative’) & (df.City == ‘London’)]
The datetime data type allows you to reformat a column in your pandas dataframe where you want to be able to handle dates, sort by oldest/recent dates or even group by week/month. This can easily be done with the to_datetime() function in pandas.
df[‘date’]=pd.to_datetime(df[‘date’], infer_datetime_format=True)
And break down the dates into component pieces if we want to get a more detailed view.
df['month'] = df['date'].dt.month
The join or merging of two dataframes can be done on columns or indexes. If joining columns on columns, the DataFrame indexes will be ignored an a new index will be created for the resulting merged dataset. Otherwise if joining the two dataframes on indexes, the index will be taken as the common column where the merging happens.
merged = pd.merge(df, df2, on='episode', how='inner')merged = pd.merge(df, df2, on='episode', how='outer')
An Inner merge, will only keep the values from each dataset that share common index or column values. The Outer merge will join the totality of the dataset and return empty values for columns and rows that cannot be found in the other dataset upon merging.
Printing the column names can be done with the columns parameter, but also with the .info() function previously mentioned.
print(df.columns)df.info()
Renaming columns can be done in a quick and easy way. This can be particularly useful after using the ‘merge’ function, where the new columns are usually labelled column_x and column_y.
df.rename(columns={"oldName1": "newName1", "oldName2": "NewName2"})
Dropping columns is also a must know function. The parameter inplace = True is too often forgotten, without it you wont see any change to your dataframe.
df.drop(columns=[‘column1’, ‘column2’], inplace = True)
This pandas function will create a hierarchy between rows given the column names passed on. This first example will group the rows by nationality and count each nationality. The result will be a dataframe with two columns: Nationality and count.
df1 = df.groupby(["nationality"]).count().reset_index()
The second example is particularly useful when don’t want to lose the information stored in the other columns. Here, the resulting dataframe will have three columns: the name, nationality and the count of names per nationality.
df1 = df.groupby(["name", "nationality"]).count().reset_index()
When visualising different groupings in a dataframe, this can be particularly useful. The following article will walk you through how to use this function for bar chart plotting.
This is by far one of the most powerful applications for every day use of pandas dataframes. The apply() function, as its name states allows us to apply a function to each row of a dataframe. The apply() method passes into the the string_to_float() function a row from the df dataframe one by one.
def string_to_float(row)    row['value'] = float(row[value'])    return row(['value'])df['price'] = df.apply(string_to_float, axis = 1)
The row[‘value’] is simply the row of the df dataframe that is being passed onto the function. I can access any of the columns in this row by specifying their name.
The lambda function is an anonymous function which is not not bound to an identifier. The general structure is the following:
lambda arguments : expression
The expression is executed on the arguments and the result is returned.
A very basic example of a lambda function applied in a non-pandas context.
x = lambda a : a + 10print(x(5))
If you wanted to apply a lambda function to an entire dataset, the syntax would be the following.
df['value'] = df['value'].apply(lambda x: float(x) if != "Not Available" else none)
This example changes the datatype of the column when the value is ‘Available’.
This is something I will leave up to you... Depending on how many tasks you want to execute a proper function might be more easy to build. While quick and simple changes can be affected using a lambda function.
For more articles like this, find me on Medium here!
If you have any questions, suggestions or ideas on how to improve, please leave a comment below or get in touch through LinkedIn here.