Package: littlebird-twitter-utils
Little BirdBasic utilties for processing Tweets. Includes:TweetTokenizerfor tokenizing the Tweet contentTweetReaderfor easily iterating over TweetsTweetWriterfor conveniently writing one or more Tweets to a file in JSONlines formatInstallationThere are two options for installinglittlebird.1. Install from sourcegitclonehttps://github.com/AADeLucia/littlebird.gitcdlittlebird
pythonsetup.pydevelop2. Install withpippipinstallgit+git://github.com/AADeLucia/littlebird.git#egg=littlebirdRead and Write a Tweet JSONlines fileThe example below reads in a Tweet file, filters to Tweets that have a hashtag, and writes out to a new file.TweetWritercan write a single Tweet or list of Tweets to a file in JSONlines format. It will also automatically open a GZIP file if the provided filename has a.gzextension. If you are writing to a GZIP file, it is recommended to write all Tweets at once instead of writing incrementally; this provides better file compression. If you do need to write incrementally, I recommend writing to a normal file and GZIPping after.fromlittlebirdimportTweetReader,TweetWriter# File in JSONlines form. Automatically handles GZIP files.tweet_file="2014_01_02.json.gz"reader=TweetReader(tweet_file)# Iterate over Tweets# Save Tweets that contain hashtagsfiltered_tweets=[]fortweetinreader.read_tweets():iftweet.get("truncated",False):num_hashtags=len(tweet["extended_tweet"]["entities"]["hashtags"])else:num_hashtags=len(tweet["entities"]["hashtags"])ifnum_hashtags>0:filtered_tweets.append(tweet)# Write out filtered Tweetswriter=TweetWriter("filtered.json")writer.write(filtered_tweets)You can also skip retweeted and quoted statusesfortweetinreader.read_tweets(skip_retweeted_and_quoted=True):Tokenize TweetsThere are multiple pre-defined tokenizers that are useful for different downstream usecases.TweetTokenizer: default customizable tokenizerBERTweetTokenizer: tokenizer based off the original BERTweet pre-processing fromthe BERTweet model repo.GloVeTweetTokenizer: tokenizer based off a modified version of theGloVepreprocessor. It's based off of a modified version because theoriginal ruby processing scriptdoes not work.You can also write your own tokenizer. Examples are below.Example usageA basic example using the default Tokenizer settings is below.fromlittlebirdimportTweetReader,TweetTokenizer# File in JSONlines form. Automatically handles GZIP files.tweet_file="2014_01_02.json.gz"reader=TweetReader(tweet_file)tokenizer=TweetTokenizer()# Iterate over tweets and get the tokenized text.# Automatically checks for retweeted and quoted text except if# tokenizer = TweetTokenizer(include_retweeted_and_quoted_content=False)fortweetinreader.read_tweets():# Tokenize the Tweet's texttokens=tokenizer.get_tokenized_tweet_text(tweet)# Get tweet hashtags. Also includes hashtags from retweeted and quoted statuseshashtags=tokenizer.get_hashtags(tweet)You can directly access the internaltokenizemethod withtokenizer.tokenize(text).If you do not need to skip over tweets and only need the text, then you can use thetokenize_tweet_file()method. This method gets the tokenized text from the entire tweet (retweet, quoted, and full text).fromlittlebirdimportTweetTokenizertweet_file="2014_01_02.json.gz"tokenizer=TweetTokenizer()# Returns a tokenized string for each tweettokenized_tweets=tokenizer.tokenize_tweet_file(tweet_file)# Returns a list of tokens for each tweettokenized_tweets=tokenizer.tokenize_tweet_file(tweet_file,return_tokens=True)Tokenizer settingsAvailableTweetTokenizersettings:language: right now it only really supports English, but as long as you change thetoken_patternaccordingly, it should work with other languages. A future integration is usingMosesfor Arabic tokenization.token_pattern: Pattern to match for acceptable tokens. Default isr"\b\w+\b"stopwords: provide a list of stopwords to remove from the text. Default isNonefor no stopword removal.remove_hashtags: Default isFalseto keep hashtags in the text (only strips the "#" symbol)lowercase: Default isTrueto lowercase all of the text. Change this toFalseif you are doing case-sensitive tasks like Name Entity Recognition (NER)The tokenizer works in the following steps:Remove hashtags (optional)Remove URLs, handles, and "RT"Lowercase the text (optional)Find all tokens that match thetoken_patternwithregex.findall(token_pattern, text)Remove stopwords (optional)Token patternsThe token pattern is extremely important to set for your use case. Below are some sample token patterns, but I highly recommendrefreshing on your regular expressionsif you need something more advanced.Note:theregexpackage is used to access character classes like\p{L}. BasicallyJava regex patterns.r"\b\w+\b"matches any token with one or more letters, numbers, and underscores. This is equivalent to"[\p{L}\_\p{N}]+"r"\b\p{L}+\b"matches any token with one or more "letters" (across all alphabets).r"\b[\w']\bmatches any token with one or more letters, numbers, and underscores. This pattern also preserves apostraphes "'", which is useful for not splitting contractions (e.g. not "can't" -> "can t"). It does not preserve quotes (e.g. "She said 'hello'" -> "she said hello")Writing your own tokenizerIf you would like to write your own, simply extend theBaseTweetTokenizerclass. See below andtweet_tokenizer.pyfor examples.fromlittlebirdimportBaseTweetTokenizerclassNewTweetTokenizer(BaseTweetTokenizer):def__init__(self,include_retweeted_and_quoted_content=True):# Initialize super classsuper().__init__(include_retweeted_and_quoted_content)# Define class variables and settingsself.token_pattern="aa*"# Overwrite the tokenize methoddeftokenize(self,tweet):regex.findall(self.token_pattern,tweet)return# Define any helper methods as neededdef_helper(self):returnThis package is a work in progress. Feel free to open any issues you run into or recommend features. I started this package for my own NLP research with tweets.@misc{DeLucia2020,
    author = {Alexandra DeLucia},
    title = {Little Bird},
    year = {2020},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/aadelucia/littlebird}},
}Copyright (c) 2020 Alexandra DeLucia