Data exploration with UNIX in Jupyter Notebook.
By: Ankur Kesharwani
Tags: Machine Learning, Python, Linux, Data Science, Open Source

Photo by rawpixel on Unsplash

Crazy about data but afraid of exploring it due to the ever confusing libraries and frameworks?

Well, you just stumbled upon on the right thing :P

Data is everywhere! from social media sites like facebook, twitter etc. to healthcare and what not! With the advent of cutting edge Machine learning and A.I technologies, the data which was of almost no significance a decade ago is now precious to the people working on these technologies and helping them predict pretty useful things like the detection of cancer at an early stage, Upcoming Weather conditions, Cyber attacks, Credit card fraud detection and what not! With the ever-increasing data in today’s world, we need to understand the data in and out to make meaningful predictions and utilise it in the best way possible for the benefit of the masses.

Jupyter Notebook has proved to be a gem for the large data exploration task due to its useful support of libraries and visualisation tools. Not just this! The Jupyter Notebook can even run your UNIX commands that you run on your terminal!

Pretty cool, right?

So, let's begin with our journey of exploring the data from the jupyter notebook itself!

To run any Unix command on the Jupyter notebook, you need to start the command with an ! sign, for example, to list all the files in your current working directory, you need to type !ls command in your cell.

Use command %env filename to store the path of the file in the variable filename for further exploration task.

%env filename = Path of the file on which operations are being performed.

head

The head command is used to display the top n lines in the file. The value of n defaults to 10 if not specified explicitly.

!head -n N $filename This command will display the top 'N' lines from the file.

tail

The tail command is used to display the last n lines in the file.

!tail -n N $filename This command will display the last 'N' lines from the file.

wc

wc stands for wordcount, it prints the number of lines, words and characters.

!wc $filename -> Prints total number of lines, words and characters.

!wc -l $filename -> Prints only the total number of lines in the file.

cat

cat displays the content of the file.

!cat $filename -> Prints out the content of the file. !cat $filename | wc -l -> Prints out the No. of lines in the file.

Note : The pipe operator | feeds the output of first command !cat $filename as the input to the second command wc-l .

grep

grep is an extremely powerful tool to look for a text in one or more files. For example in the next command we are looking for all the lines that contain a word, we also specify with -i that we are interested in case insensitive matching, i.e. we don't care about case.

!grep -i 'Dawn' $filename -> Prints all the lines containing the

string 'Dawn' in it.

!grep -i 'Dawn' $filename | wc -l -> Prints the count of all lines having string 'Dawn' in it.

sed

sed is a powerful stream editor, it works similarly to grep , but it also modifies the output text. It uses regular expressions, which are a language to define pattern matching and replacement.

Syntax : !sed -e 's/from/to/g' $filename

here,

s stands for substitution

stands for substitution from is the word to be matched

is the word to be matched to is the replacement string

is the replacement string g specifies to apply this to all occurrences on a line, not just the first.

!sed -e s/x/y/g $filename ->This will replace all occurences of x with y.

So, this was it for the exploration part in the Jupyter Notebook! Hope you guys learnt something new today.

About me :

I am Ankur Kesharwani, 3rd year Undergrad at JIIT Noida, currently exploring Open source and Data science. Connect with me on LinkedIn and Github.