How to Tokenize Tweets with Python | by Khuyen Tran | Towards Data Science

A token is a piece of a whole, so a word is a token in a sentence, and a sentence is a token in a paragraph. Tokenization is the process of splitting a string into a list of tokens.
If you are somewhat familiar with tokenization but don’t know which tokenization to use for your text, this article will use raw Tweets from Twitter to show different tokenizations and how they work.
This article will cover how to tokenize sentence into words with:
word_tokenize
WordPunctTokenizer
RegrexTokenizer
TweetTokenizer
Tokenization is one of the first steps to preprocess a raw text, so I hope you are excited to master this important concept!
Twitter is a social platform that many interesting tweets are posted every day. Because tweets are more difficult to tokenize compared to formal text, we will use the text data from tweets as our example.
"https://t.co/9z2J3P33Uc FB needs to hurry up and add a laugh/cry button 😬😭😓🤢🙄😱 Since eating my feelings has not fixed the world's problems, I guess I'll try to sleep... HOLY CRAP: DeVos questionnaire appears to include passages from uncited sources https://t.co/FNRoOlfw9s well played, Senator Murray Keep the pressure on: https://t.co/4hfOsmdk0l @datageneral thx Mr Taussig It's interesting how many people contact me about applying for a PhD and don't spell my name right."
There are a lot of pieces of information on the sentence above. Before tokenizing the whole sentence, let’s pick some sentences that we are interested in comparing. This list will be used to compare the performance between different tokenizers.
compare_list = ['https://t.co/9z2J3P33Uc',               'laugh/cry',               '😬😭😓🤢🙄😱',               "world's problems",               "@datageneral",                "It's interesting",               "don't spell my name right",               'all-nighter']
The most popular method when tokenizing sentences into words is word_tokenize. word_tokenize separate words using spaces and punctuations.
from nltk.tokenize import word_tokenizeword_tokens = []for sent in compare_list:    print(word_tokenize(sent))    word_tokens.append(word_tokenize(sent))
Outcome:
['https', ':', '//t.co/9z2J3P33Uc']['laugh/cry']['😬😭😓🤢🙄😱']['world', "'s", 'problems']['@', 'datageneral']['It', "'s", 'interesting']['do', "n't", 'spell', 'my', 'name', 'right']['all-nighter']
We want laugh/cry is split into 2 words. So we should consider another tokenizer option.
WordPunctTokenizer splits all punctuations into separate tokens. So this may be what we want?
from nltk.tokenize import WordPunctTokenizerpunct_tokenizer = WordPunctTokenizer()punct_tokens = []for sent in compare_list:    print(punct_tokenizer.tokenize(sent))    punct_tokens.append(punct_tokenizer.tokenize(sent))
Outcome:
['https', '://', 't', '.', 'co', '/', '9z2J3P33Uc']['laugh', '/', 'cry']['😬😭😓🤢🙄😱']['world', "'", 's', 'problems']['@', 'datageneral']['It', "'", 's', 'interesting']['don', "'", 't', 'spell', 'my', 'name', 'right']['all', '-', 'nighter']
Hmm, this tokenizer successfully splits laugh/cry into 2 words. But the fallbacks are:
The link ‘https://t.co/9z2J3P33Uc' is split into 7 words
world's is split into 2 words by "'" character
@datageneral is split into @ and datageneral
don't is split into do and n't
Since these words should be considered as one word, this tokenizer is not what we want either. Is there a way that we can split words based on the space instead?
Since there is not the tokenizer specifically splitting up words based on the space, we can instead use RegrexTokenizer to control how to tokenize text. There are two ways that we can avoid split up words based on punctuations or contractions:
Match on the tokens
Match on the separators or gaps
The RegexpTokenizer class works by compiling our pattern, then calling re.findall()on our text. We could utilize this function to match alphanumeric tokens plus single quotes
from nltk.tokenize import RegexpTokenizermatch_tokenizer = RegexpTokenizer("[\w']+")match_tokens = []for sent in compare_list:       print(match_tokenizer.tokenize(sent))    match_tokens.append(match_tokenizer.tokenize(sent))
If you are not familiar with regex syntax, \w+ matches one or more word character (alphanumeric & underscore)
Outcome:
['https', 't', 'co', '9z2J3P33Uc']['laugh', 'cry'][]["world's", 'problems']['datageneral']["It's", 'interesting']["don't", 'spell', 'my', 'name', 'right']['all', 'nighter']
While words such as 'world’s', 'It’s', 'don’t’ are kept as one entity as we want, ‘https://t.co/9z2J3P33Uc' is still split into different words and we lose the “@” character before “datageneral”. Maybe we could split based on whitespace instead?
RegexpTokenizer can also work by matching the gaps. When the parameter gaps=True is added, the matching pattern will be used as the separators. \s+ matches one or more space.
space_tokenizer = RegexpTokenizer("\s+", gaps=True)space_tokens = []for sent in compare_list:        print(space_tokenizer.tokenize(sent))    space_tokens.append(space_tokenizer.tokenize(sent))
Outcome:
['https://t.co/9z2J3P33Uc']['laugh/cry']['😬😭😓🤢🙄😱']["world's", 'problems']['@datageneral']["It's", 'interesting']["don't", 'spell', 'my', 'name', 'right']['all-nighter']
Nice! Now we have the link ‘https://t.co/9z2J3P33Uc' interpreted as one word! But it seems like the emojis are grouped as one word. Since different emojis can be meaningful in sentiment analysis, we might want to split them into different words. So we need to contemplate another regex pattern that enables us to do that?
Good news! There is a tokenizer that can split tweets efficiently without using regex.
Yes, the best way to tokenize tweets is to use the tokenizer built to tokenize tweets
from nltk.tokenize import TweetTokenizertweet_tokenizer = TweetTokenizer()tweet_tokens = []for sent in compare_list:    print(tweet_tokenizer.tokenize(sent))    tweet_tokens.append(tweet_tokenizer.tokenize(sent))
Outcome:
['https://t.co/9z2J3P33Uc']['laugh', '/', 'cry']['😬', '😭', '😓', '🤢', '🙄', '😱']["world's", 'problems']['@datageneral']["It's", 'interesting']["don't", 'spell', 'my', 'name', 'right']['all-nighter']
Awesome! The tweets are tokenized exactly like how we want!
Instead of taking the time to analyze the outcome of each tokenizer, we can put everything in one pd.dataframe for fast and accurate interpretation.
import pandas as pdtokenizers = {'word_tokenize': word_tokens,             'WordPunctTokenize':punct_tokens,             'RegrexTokenizer for matching':match_tokens,             'RegrexTokenizer for white space': space_tokens,             'TweetTokenizer': tweet_tokens }df = pd.DataFrame.from_dict(tokenizers)
From the observation of the table above, TweetTokenizer seems like the optimal choice. So we could go ahead and use this to tokenize our sentence:
tweet_tokenizer.tokenize(sent)
Congratulation! You have learned different tokenizers from nltk library to tokenize sentences into words. It seems like the winner in tokenizing the Twitter raw text is TweetTokenizer . But this is not always the case, your pick may change depending on the text you analyze. The important point is that you know the difference in the functionality of these tokenizers so that you could make the right choice for tokenizing your text. Feel free to fork and play with the code for this article in this Github repo.
I like to write about basic data science concepts and play with different algorithms and data science tools. You could connect with me on LinkedIn and Twitter.
Star this repo if you want to check out the codes for all of the articles I have written. Follow me on Medium to stay informed with my latest data science articles like these: