Data Handling From Different File Types And Basic Database Manipulation Using Python
By: Abhijit Tripathy
Tags: Python Programming, Scikit Learn, Python, Data Science, Pandas Dataframe

Hello learners. In this article, I am going to discuss three basic problem statements that can be helpful in your “Data Science” journey. This article holds the methods to handle data and load data from different file types like CSV, EXCEL, JSON and about querying simple SQL database using Python.

Problem Statement 1:-

Suppose, you have a machine learning model ready and you want some ready-made data set to train and test our model before deploying.

Soln:

The “scikit-learn” library of python provides the user with three sample datasets, available as preloaded sets inside the library.

load_digits() — This data set is used mostly for image classification models. load_boston() — This is used for linear regression, multiple regression and other regression problems. load_iris() — Classification problems can be tested best on this data set as it provides some labelled cluster types as data.

loading readymade datasets from Scikit-Learn

Here, dataset is the module of sklearn library that holds all the pre-loaded datasets. Hence we need to import it from sklearn first.

feature and target extraction from the loaded data

Similarly, we can load the other two data sets as follow;

loading boston and iris data by using scikit-learn

One crucial and useful thing in the pre-existing data set is that we don’t need to clean the data set and make it ready manually. The data is available as a ready to use set, which in term allows us to skip the data wrangling processes before training our Machine Learning model.

Problem Statement 2:

how to use a CSV(Comma-Separated Value), Excel, JSON file data as the input data in your model?

Soln:

The pandas library of python gives enormous freedom and control to the programmers. You can use its built-in functions to load the files as a python dataframe. “Dataframe” in python is just like a table having rows and columns, which holds data and provide many functions that can help us to manipulate the table elements and make the data table ready to use.

Loading different type of files using pandas dataframe

Problem Statement 3:

How to load data from an SQLite database into your python snippet, which means how to connect database with the python code?

Soln:

There are many types of databases like MySQL, PostgreSQL and SQLite etc. Among these SQLite comes with a database browser and very easy to operate. Let’s have a look at how to query from a SQL database.

Using the SQL query to perform operation on databases

So basically what happens here is, we make a connection of our python code with our SQLite database first by using .connect() and then we pass the required query “SELECT * FROM Album” as an argument along with the python object conn. The python compiler passes the command to the SQLlie editor and makes it execute, collects the output data of the database query and stores it as a dataframe data4.

So, what is an SQL query?

This is just like a command we give to our machine through the command line. Here we give this command known as a query to the SQL database. “Album” is one of the tables in the database and by using SELECT * you are passing a command to the database to select all the rows of the table “Album” and return the output/ print them in the console.

Loading the data to a data frame object is the first step we perform while creating a machine learning program or model.

That’s all for today guys, please subscribe, share and comment.

Thank you.