Pandas concat() tricks you should know to speed up your data analysis | by B. Chen | Towards Data Science

Pandas provides various built-in functions for easily combining DataFrames. Among them, the concat() function seems fairly straightforward to use, but there are still many tricks you should know to speed up your data analysis.
In this article, you’ll learn Pandas concat() tricks to deal with the following common problems:
Dealing with index and axisAvoiding duplicate indicesAdding a hierarchical index with keys and names optionsColumn matching and sortingLoading and concatenating datasets from a bunch of CSV files
Dealing with index and axis
Avoiding duplicate indices
Adding a hierarchical index with keys and names options
Column matching and sorting
Loading and concatenating datasets from a bunch of CSV files
Please check out my Github repo for the source code.
Suppose we have 2 datasets about exam grades.
df1 = pd.DataFrame({    'name': ['A', 'B', 'C', 'D'],    'math': [60,89,82,70],    'physics': [66,95,83,66],    'chemistry': [61,91,77,70]})df2 = pd.DataFrame({    'name': ['E', 'F', 'G', 'H'],    'math': [66,95,83,66],    'physics': [60,89,82,70],    'chemistry': [90,81,78,90]})
The simplest concatenation with concat() is by passing a list of DataFrames, for example[df1, df2]. And by default, it is concatenating vertically along the axis 0 and preserving all existing indices.
pd.concat([df1, df2])
If you want the concatenation to ignore existing indices, you can set the argument ignore_index=True. Then, the resulting DataFrame index will be labeled with 0, ..., n-1.
pd.concat([df1, df2], ignore_index=True)
To concatenate DataFrames horizontally along the axis 1 , you can set the argument axis=1 .
pd.concat([df1, df2], axis=1)
Now, we know that the concat() function preserves indices. If you’d like to verify that the indices in the result of pd.concat() do not overlap, you can set the argument verify_integrity=True. With this set to True, it will raise an exception if there are duplicate indices.
try:    pd.concat([df1,df2], verify_integrity=True)except ValueError as e:    print('ValueError', e)ValueError: Indexes have overlapping values: Int64Index([0, 1, 2, 3], dtype='int64')
It is quite useful to add a hierarchical index (Also known as multi-level index) for more sophisticated data analysis. In this case, let’s add index Year 1 and Year 2 for df1 and df2 respectively. To do that, we can simply specify the keys argument.
res = pd.concat([df1, df2], keys=['Year 1','Year 2'])res
And to access a specific group of values, for example, Year 1:
res.loc['Year 1']
In addition, the argument names can be used to add names for the resulting hierarchical index. For example: add name Class to the outermost index we just created.
pd.concat(    [df1, df2],     keys=['Year 1', 'Year 2'],    names=['Class', None],)
To reset an index and turn it into a data column, you can use reset_index()
pd.concat(    [df1, df2],     keys=['Year 1', 'Year 2'],    names=['Class', None],).reset_index(level=0)   # reset_index(level='Class')
The concat() function is able to concatenate DataFrames with the columns in a different order. By default, the resulting DataFrame would have the same sorting as the first DataFrame. For example, in the following example, it’s the same order as df1.
If you prefer the resulting DataFrame to be sorted alphabetically, you can set the argument sort=True.
pd.concat([df1, df2], sort=True)
If you prefer a custom sort, here is how to do it:
custom_sort = ['math', 'chemistry', 'physics', 'name']res = pd.concat([df1, df2])res[custom_sort]
Suppose we need to load and concatenate datasets from a bunch of CSV files. Here is one solution using for loop.
# Badimport pathlib2 as pl2ps = pl2.Path('data/sp3')res = Nonefor p in ps.glob('*.csv'):    if res is None:        res = pd.read_csv(p)    else:        res = pd.concat([res, pd.read_csv(p)])
This certainly does the work. But the pd.concat() gets called every time in each for loop iteration. We can solve this effectively using list comprehension.
import pathlib2 as pl2ps = pl2.Path('data/sp3')dfs = (    pd.read_csv(p, encoding='utf8') for p in ps.glob('*.csv'))res = pd.concat(dfs)res
A single line of code read all the CSV files and generate a list of DataFrames dfs. Then, we just need to call pd.concat(dfs) once to get the same result.
If you time both executions using %%timeit, you probably find that the list comprehension solution saves half of the time.
# for-loop solution298 ms ± 11.8 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)# list comprehension solution153 ms ± 6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
List comprehension saves time and codes. It is a simple way to generate a list comparing to using loops.
Thanks for reading.
Please check out the notebook for the source code.
Stay tuned if you are interested in the practical aspect of machine learning.
How to do a Custom Sort on Pandas DataFrame
When to use Pandas transform() function
Using Pandas method chaining to improve code readability
Working with datetime in Pandas DataFrame
Working with missing values in Pandas
Pandas read_csv() tricks you should know
4 tricks you should know to parse date columns with Pandas read_csv()
More can be found from my Github