7 things to quickly improve your Data Analysis in Python | by Peter Nistrup | Towards Data Science

Take your Data Analysis to the next level!
Turning a blind eye to the completely obvious risk of sounding like a broken record I just want to voice yet another giant thank you to everyone who’s been reading and sharing my last two articles: Python tricks 101, what every new programmer should know and Exploring your data with just 1 line of Python. Here I was, thinking that the “Python tricks 101...” article had been successful and then you go ahead and blow any expectations away once more.So, to quote myself:
“Thanks and lets get on with it!”
For this article I thought it would be nice to create a list of things I’ve learned that have sped up or improved my average day-to-day data analysis. So without any further ado, here’s a list of what we’ll be covering in the article:
Pandas ProfilingPlotting Pandas data using Cufflinks & PlotlyIPython Magic CommandsFancy formatting in JupyterJupyter shortcutsMultiple outputs per cell in Jupyter (or IPython)Instantly create slideshows of your Jupyter Notebook
Pandas Profiling
Plotting Pandas data using Cufflinks & Plotly
IPython Magic Commands
Fancy formatting in Jupyter
Jupyter shortcuts
Multiple outputs per cell in Jupyter (or IPython)
Instantly create slideshows of your Jupyter Notebook
If these sound like something you might enjoy learning about, keep scrolling.
Note that this will be quite a long one. Still, I’ve tried my best to be as concise and as to-the-point as possible. I hope you enjoy!
This should be no surprise to those of you who’ve been following me. The benefits of this tool are obvious. The animation below is the results of calling the simple method df.profile_report(), take a look at the resulting report yourself:
Using this tool is simple a matter of installing and importing the Pandas Profiling package.
I wont spend too much time on this one since I’ve already written a small article on the subject, so if you want to know more about this gem of a package, then please read my article about Pandas Profiling here:
towardsdatascience.com
Most of us who’re quote-unquote “experienced” Data Scientist or Data Analysts, will be quite familiar with the integration between matplotlib and pandas. Namely the fact that you can make quick plots of a simple pd.DataFrame or pd.Series by simply calling the .plot() method:
Now this is all fine and dandy, but how about an interactive, pan-able, zoom-able, scale-able plot instead? Cufflinks to the rescue! (courtesy of Plotly)
To install Cufflinks in your environment simply run ! pip install cufflinks --upgrade in your favorite terminal and you’re ready to go. Take a look at the following:
Notice that the only thing that changed above is the imports and the setting of cf.go_offline() for Cufflinks. Other than that it’s simply a matter of changing your .plot() method to .iplot() .
Other methods like .scatter_matrix() can give you some really nice visualisation as well:
I urge anyone that works with a lot of data visualisation to take a look at the documentation for both Cufflinks and Plotly to discover more methods for yourselves!
The ‘Magics’ of IPython are basically a series of enhancements that IPython has layered on-top of the standard Python syntax. Magic commands come in two flavors: line magics, which are denoted by a single % prefix and operate on a single line of input, and cell magics, which are denoted by a double %% prefix and operate on multiple lines of input. Below are just a few of the useful functionalities provided by the ‘Magics’:
Why not start with some good-old self-help? If you’re to remember just one Magic Command it should probably be this one. Executing the %lsmagic command will grant you a full list of all available Magic Commands:
This is probably the Magic Command I use the most.
Most of us have tried it: You find yourself executing a code-block that keeps breaking and you — in pure desperation — write 20 print()-statements to output the contents of each variable. Then, when you finally fix the issue, you have to go back and delete all the printing functionality again.
But not anymore! Simply execute the %debug command after encountering a problem and execute whatever part of the code you want:
So what’s happening above?
We have a function that takes a list as input and squares all even number.We run our function but something breaks. We’re not told at what!We use the %debug command to tab into the function at the time it broke.We ask the debugger to tell us the value of x as well as the type(x).The failure is obvious. We’re passing '6' as a string!
We have a function that takes a list as input and squares all even number.
We run our function but something breaks. We’re not told at what!
We use the %debug command to tab into the function at the time it broke.
We ask the debugger to tell us the value of x as well as the type(x).
The failure is obvious. We’re passing '6' as a string!
This is highly useful for more complex functions as I’m sure you can imagine!
This one is pretty cool as well. Let’s say for the sake of argument, that you spend a while cleaning some data in a notebook, now you want to test some functionality in another notebook so you have to either implement the functionality in the same notebook or save your data and load it in the other notebook right? Not with %store ! Simply store the variable and retrieve it in whatever other notebook you want:
%store [variable] stores the variable.%store -r [variable] reads/retrieves a stored variable.
Ever assigned some value to a variable but forgot the variable name? Maybe you accidentally deleted the cells responsibility for assigning the value to the variable? With %who you get a list of all globally assigned variables:
With this little command you get all the timing information you might need. Simply apply the %%time command to any piece of executable code you want and you’ll get an output like this:
And this will be the last one I’ll talk about in this article. This one is especially useful for when you’ve written some complicated function or class in your notebook that you want to save in its own file, separate from the notebook. Simply prefix the cell of your function or class with the %%writefile command followed by a filename to save it:
As you can see, we can simply save the function we’ve created to a “utils.py” file and then we’re free to import it in whatever fashion we want. In other notebooks as well! (As long as they’re in the same directory of the utils.py file)
This one is super cool! Basically Jupyter allows for some HTML / CSS formatting in the markdown cells. These are the once I use the most:
The blue and stylish one:
<div class="alert alert-block alert-info">    This is <b>fancy</b>!</div>
The red and slightly panic-inducing one:
<div class="alert alert-block alert-danger">    This is <b>baaaaad</b>!</div>
The green and calming one:
<div class="alert alert-block alert-success">    This is <b>gooood</b>!</div>
Take a look at them in action here:
These are quite nice for when you want to present some findings in a Notebook format!
Accessing and learning the keyboard shortcuts, you can use the command palette: Ctrl + Shift + P . This brings up a list of all functionalities of the notebook. The following is an excerpt of the most basic commands:
Esc : This will take you into command mode. This is also the mode in which you can navigate your notebook using the arrow-keys.
While in command mode:
A and B : Inserts a new cell [A]bove or [B]elow the current cell.
M : Changes the current cell to [M]arkdown.
Y : Changes the current cell to code. (y tho?)
D,D : [D]efinetly [D]eletes the current cell.
Enter : Takes you back into edit mode for the current cell.
While in edit mode:
Shift + Tab : Shows you the Docstring (documentation) for the the object you typed in the current cell — keep pressing the shortcut to cycle through modes of documentation.
Ctrl + Shift + - : Splits the current cell from where your cursor is.
Esc + F : Find and replace on your code excluding outputs.
Esc + O : Toggles cell output.
Selecting Multiple Cells:
Shift + Down and Shift + Up : Selects the next sell in a downwards or upwards direction. I think you can use the power of deduction to find out which is which.
Shift + M : Merges the selected cells.
As a side-note, remember that once cells are selected, you can delete / copy / cut / paste / run them as a batch.
This one is great. Have you ever wanted to display the .head() and the .tail() of a pandas DataFrame but you you gave up halfway-through because it was too much of a hassle to create the additional code-cell to run the .tail() method? Fear no more, now you can display all the outputs you want using the following lines of code:
from IPython.core.interactiveshell import InteractiveShellInteractiveShell.ast_node_interactivity = "all"
Now lo and behold, the power of multiple outputs:
With RISE you can instantly turn your Jupyter Notebook into a slideshow with a single keystroke. Best of all, the notebook is still active, so you can perform live coding while presenting! Take a look:
To use this brilliant tool, simple install RISE either by virtue of conda or pip depending on your environment:
conda install -c conda-forge rise/// OR ///pip install RISE
Now you’ll be able to create fancy slides from your notebook by simply clicking the new button:
Thanks for reading. I would urge anyone to explore each point in more detail on your own to find out exactly what and how you want to apply them to your own workflows.
If you want to see and learn more, be sure to follow me on Medium🔍 and Twitter 🐦