explore the dataset, obtain the data
common steps in natural language processing
data acquisition and cleaning
cleaning, sampling and sentence tokenization
tokenization using NGramTokenizer
profanity filtering, using google list of swearing words
exploratory analysis, questions about frequencies
Zipf's law, frequency distribution
bigrams frequency distribution, for all 4 languages and 3 media types
vector graphics of zipf's law and bigrams frequency distribution
trigram frequency distribution per corpus language and media type
unique words that cover half of the word instances
create a table for the results of the unique words that cover half of the word instances
calculate and report the results for the unique words needed to cover the 90% of the word instances