Open Source Dataset for NLP Beginners
By: Dr. Monica
Tags: Machine Learning, Python, Data Science, NLP, Artificial Intelligence

Nowadays, Natural language processing is an up growing and booming field of research, . Sometimes, it might be very confusing for an NLP beginner, which areas to explore and how to find the exact dataset for the implementation and hands-on experience.

This blog is focused to provide an overview of different free online datasets for NLP. It is quite difficult to compile all the datasets, as NLP is a broad range research area but keeping respective of a beginner in mind and general problem statements to be implemented at the starting point, I tried to build the following list.

Datasets for Sentiment Analysis

Applying Machine Learning for the Sentiment analysis task needs a large number of specialized datasets.

The following list should hint at some of the ways that you can improve your sentiment analysis algorithm.

Multidomain Sentiment Analysis Dataset: This dataset contains the features of a variety of product reviews taken from Amazon.

IMDB Reviews: This is relatively small dataset was compiled primarily for binary sentiment classification use cases which contain 25,000 movie reviews.

Stanford Sentiment Treebank: Also built from movie reviews, Stanford’s dataset was designed to train a model to identify sentiment in longer phrases. It contains over 10,000 snippets taken from Rotten Tomatoes.

Sentiment140: This dataset consists of 160,000 tweets formatted with 6 fields: polarity, ID, tweet date, query, user, and the text. Emoticons have been pre-removed.

Twitter US Airline Sentiment: This dataset contains tweets about US airlines that are classified as positive, negative, and neutral. Negative tweets have also been categorized by reason for complaint.

Text Datasets

Natural language processing is a massive field of research, but the following list includes a broad range of datasets for different natural language processing tasks, such as voice recognition and chatbots.

20 Newsgroups: This dataset is a collection of approximately 20,000 documents covers 20 different newsgroups, from baseball to religion.

ArXiv: This repository contains all of the arXiv research paper archive as full text, with a total dataset size of 270 GB.

Reuters News Dataset: The documents in this dataset appeared on Reuters in 1987. They have since been assembled and indexed for use in machine learning.

The WikiQA Corpus: This corpus is a publicly-available collection of questions and answers pairs. It was originally assembled for use in research on open-domain question answering.

UCI’s Spambase: This large spam email dataset is useful for developing personalized spam filters,created by a team at Hewlett-Packard,

Yelp Reviews: This open dataset released by Yelp contains more than 5 million reviews.

WordNet: Compiled by researchers at Princeton University, WordNet is essentially a large lexical database of English ‘synsets’, or groups of synonyms that each describe a different, distinct concept.

The Blog Authorship Corpus — This dataset includes over 681,000 posts written by 19,320 different bloggers. In total, there are over 140 million words within the corpus.

General — Datasets for Natural Language Processing

There are a few more datasets for natural language processing tasks that are commonly used in general.

Enron Dataset: This contains, roughly 500,000 messages from the senior management of Enron. This dataset is generally used by people who are are looking to improve or understand current email tools.

Amazon Reviews: This dataset contains around 35 million reviews from Amazon spanning a period of 18 years. It includes product and user information, ratings, and plaintext review.

Google Books Ngrams: A Google Books corpora of n-grams, or ‘fixed size tuples of items’, can be found at this link. The ’n’ in ‘n-grams’ specifies the number of words or characters in that specific tuple.

Blogger Corpus: This is a collection of 681,288 blog posts contains over 140 million words. Each blog included here contains at least 200 occurrences of common English words.

Wikipedia Links Data: Containing approximately 13 million documents, this dataset by Google consists of web pages that contain at least one hyperlink pointing to English Wikipedia. Each Wikipedia page is treated as an entity, while the anchor text of the link represents a mention of that entity.

Gutenberg eBooks List: This annotated list of ebooks from Project Gutenberg contains basic information about each eBook, organized by year.

Hansards Text Chunks of Canadian Parliament: This corpus contains 1.3 million pairs of aligned text chunks from the records of the 36th Canadian Parliament.

Jeopardy: The archive linked here contains more than 200,000 questions and answers from the quiz show Jeopardy. Each data point also contains a range of other information, including the category of the question, show number, and air date.

SMS Spam Collection in English: This dataset consists of 5,574 English SMS messages that have been tagged as either legitimate or spam. 425 of the texts are spam messages that were manually extracted from the Grumbletext website.

Depending on the problem statement you are aspiring to solve download the respective dataset and get explored more of yourself into world of NLP.