Best Data Sources for Data Scientists
By: Saurav Anand
Tags: Data Analysis, Artificial Intelligence, Computer Vision, Data Science, Machine Learning

Database is the information you loose when your memory crashes — Dave Barry

Introduction

According to Wikipedia, a dataset or data set is collection of data. In the open data discipline , the dataset is the unit to measure the information released in a public open data repository. The most common format for datasets we will find online are in the form of csv and spreadsheets where the data is organized in tabular form. In the case of tabular data, a data set corresponds to one or more database tables , where every column of a table represents a particular variable, and each row corresponds to a given record of the data set in question. The data set lists values for each of the variables, such as height and weight of an object, for each member of the data set. Each value is known as a datum. Data sets can also consist of a collection of documents or files.

The data present in the dataset can be in the form of images, videos, audio files, numerical data or textual data and are stored in different formats. It is not necessary that there should be only one file , the dataset can be in the form of zip file or folder containing multiple data tables with related data.

How the Datasets are created ?

The datasets are created in multiple ways. They have been collected through surveys. Some of the data are recorded from human observation. Data can also be scrapped from websites or pulled via API’s. Even data can also be machine generated data.

It’s always important to understand that how was this dataset created ? Where does this dataset comes from? It’s always recommended to understand the data we are working from with.

Where to find datasets ?

Some of the most commonly used datasets sources are listed below and are used by the data scientists :-

1. Kaggle

Kaggle, a subsidiary of Google LLC, is an online community of data scientists and machine learning practitioners. It is a place where you can learn, practice and fine tune our data science and analytics skills. There are lot of open , public data and allow users of the platform to share codes so that we can learn best practices within the data space.

Link :- https://www.kaggle.com/datasets

2. UCI Machine Learning Repository

University of California Irvine hosts 440 data set as a service to the ML community. These data sets are good , cleaned and can be used for the analytics and modelling purpose.

Link: http://archive.ics.uci.edu/ml/index.php

3. Google Public datasets

Google Cloud Public Datasets facilitate access to high-demand public datasets, making it easy for you to access and uncover new insights in the cloud. By analyzing these datasets hosted in BigQuery and Cloud Storage, you can seamlessly experience the full value of Google Cloud with ease.

Google lists all the datasets on a page. On google cloud platform (GCP) , you can query using BigQuery to explore the datasets. We will need to sign up for the GCP account

Link: cloud.google.com/bigquery/public-data/

4. FiveThirtyEight

FiveThirtyEight, sometimes rendered as 538, is a website that focuses on opinion poll analysis, politics, economics, and sports blogging. It is an interactive news and sports site that has some exceptional data visualizations. They make lot of data in open for the public view, meaning that we can download and play with the data ourself

Link: https://data.fivethirtyeight.com/

5. Buzzfeed News

It makes the data sets, analysis , libraries , tools , and guides used in it’s article available on Github. Check them out to learn from some of the best

Link: github.com/BuzzFeedNews

6. data.world

data.world’s cloud-native data catalog makes it easy for everyone — not just the “data people” — to get clear, accurate, fast answers to any business question.A data catalog is a metadata management tool that companies use to inventory and organize the data within their systems. Typical benefits include improvements to data discovery, governance, and access.

Link:https://data.world/

7. Socrata

Socrata hosts cleaned open source data sources ranging from the government , business, and education data sets. The Socrata Open Data API allows you to programmatically access a wealth of open data resources

Link: https://opendata.socrata.com/

8. Awesome public datasets

This Github hosts a library of awesome, public datasets . They are sorted by category and link us straight to the hosting website.

Link: github.com/awesomedata/

9. Quandl

Quandl is a marketplace for financial, economic and alternative data delivered in modern formats for today’s analysts, including Python, Excel, Matlab, and R. Some of the datasets are available for free and some can be purchased.

Link: https://www.quandl.com/search

10. Data.gov

It allows us to download and explore data from US government agencies. Data can range from government budgets to climate data. The data is very well documented so it becomes easy to navigate through the resources.

Link: https://www.data.gov/

11. Academic Torrents

It is a site that is geared around sharing the datasets from scientific papers. We can browse the data sets directly from the website and can also download it .

Link: http://academictorrents.com/browse.php

12. AWS Public Data sets

Amazon has a page that lists all the data sets for us to browse. We need an aws account for this and amazon also gives free access tier for the new accounts.

Link: https://aws.amazon.com/datasets

Some other repositories

Jeremy Singer-Vine

Wikipedia Data Sets

World Bank Data Sets

Reddit — /r/datasets

NASA Datasets

Twitter Dataset via Twitter API

Github Dataset via Github API

CERN Open Data Portal

Global Health Observatory Data Repository

References :

— Machine Learning India

— 11 websites to find free, interesting datasets: interviewqs.com.

— 21 Places to Find Free Datasets for Data Science Projects: www.dataquest.io.

Hope you liked this information

Thanks ,

Saurav Anand