Package: twitter-analysis-tools
twitter_analysis_toolsTools for loading, manipulating and filtering tweets.For analyzing twitter data.InstallationTo install twitter_analysis_tools, run this command in your terminal:$pipinstall-Utwitter-analysis-toolsThis is the preferred method to install twitter_analysis_tools, as it will always install the most recent stable release.If you don't havepipinstalled, theseinstallation instructionscan guide
you through the process.CitingIf you use our work in an academic setting, please cite our paper:UsageTo load tweets from files, the tweets are assumed to be saved as compressed json lines (file typejsonl.gz).We highlight several generally useful tools here.PipelinesThe Pipeline class of pipeline.py is the primary interface for working with tweets and derived information. Each instance of the pipeline class has a data stream and steps to be applied to that data stream.The filetwitter_analysis_tools.twitter.common_pipelines.pycontains pre built pipelines for convenience includingget_tweet_text_pipeline: from a generator of filepaths, return a pipeline that yields text from the tweets contained in those files.get_bag_of_words_per_file: form a pipeline of bag_of_words representaions of tweets for each file in filepaths.Imports and example dataThe following will be used for the remaining examples.>>>fromtwitter_analysis_tools.twitterimportget_tweets,tweet_info,common_pipelines>>>fromtwitter_analysis_tools.utilsimportPipeline>>>fromfunctionalimportseq>>>texts=["Hello my friend, how are you?","My dog ate my homework. Sorry it is late."]>>>filepaths=["test_tweets1.jsonl.gz","test_tweets2.jsonl.gz"]>>>fortext,filepathinzip(texts,filepaths):...tweet={"full_text":text,"lang":"en"}...seq([tweet]).to_jsonl(filepath,'wb',compression='gzip')Getting tweets from filesThe following code creates a pipeline of tweets from the files indicated in filepaths.>>>tweets=get_tweets.TweetsFromFiles(*filepaths)>>>list(tweets)[{'full_text':'Hello my friend, how are you?','lang':'en'},{'full_text':'My dog ate my homework. Sorry it is late.','lang':'en'}]Alternatively, we can have a stream of filepaths and get the tweets for each file. This is useful if we want to keep the tweets from each file separate.>>>tweet_files_pipeline=Pipeline(filepaths,precompute_len=True)>>>tweet_files_pipeline=tweet_files_pipeline.add_map(get_tweets.TweetsFromFiles)>>>fortweet_fileintweet_files_pipeline:...print(list(tweet_file))[{'full_text':'Hello my friend, how are you?','lang':'en'}][{'full_text':'My dog ate my homework. Sorry it is late.','lang':'en'}]Getting text from tweetsGiven a tweet object, the following will extract the text from the tweet>>>tweet_info.get_full_text({'full_text':'Hello my friend, how are you?'})'Hello my friend, how are you?'Given a generatortweetsof tweets, the following will return a generator that extract the text from each tweet, filter out english tweets, remove links, user tags and optionally exclude retweets>>>tweets=[{'full_text':'Hello my friend, how are you?','lang':'en'},{'full_text':'My dog ate my homework. Sorry it is late.','lang':'en'}]>>>list(get_tweets.text_from_tweets(tweets,include_retweets=True))['Hello my friend, how are you?','My dog ate my homework. Sorry it is late.']One can also get a pipeline of tweet text from a list of filepaths>>>list(common_pipelines.get_tweet_text_pipeline(*filepaths,include_retweets=True))['Hello my friend, how are you?','My dog ate my homework. Sorry it is late.']DocumentationTODO: readthedocs
For more information, read the docs.DevelopmentSeeCONTRIBUTING.mdfor information related to developing the code.Suggested Git Branch Strategymasteris for the most up-to-date development, very rarely should you directly commit to this branch. Your day-to-day work should exist on branches separate frommaster. It is recommended to commit to development branches and make pull requests to master.3. Even if it is just yourself working on the repository, make a pull request from your working branch tomasterso that you can ensure your commits don't break the development head. GitHub Actions will run on every push to any branch or any pull request from any branch to any other branch.4. It is recommended to use "Squash and Merge" commits when committing PR's. It makes each set of changes tomasteratomic and as a side effect naturally encourages small well defined PR's.Additional Optional Setup Steps:Create an initial release to test.PyPI and PyPI.FollowThis PyPA tutorial, starting from the "Generating distribution archives" section.Create a blank github repository (without a README or .gitignore) and push the code to it.Create an account oncodecov.ioand link it with your GitHub account. Code coverage should be updated automatically when you commit tomaster.Add branch protections tomasterGo to yourGitHub repository's settings and under theBranchestab, clickAdd ruleand select the
settings you believe best.Recommendations:Require status checks to pass before mergingSetup readthedocs. Create an account onreadthedocs.organd link it to your GitHub account.Go to your account page and select "Import a Project".Select the desired GitHub repository from the list, refreshing first if it is not present.Go to the admin panel of the new project and make some changes to the "advanced settings":Enable "Show version warning"Enter "rtd-reqs.txt" into the "Requirements file" fieldEnable "Install Project"Enable "Use system packages"Make sure to click save at the bottom when you are finished editing the settingsGo to the admin panel and find the "Automation Rules" tab.Add a new Rule called "Publish releases"Set the Version type to "Tag"Set the Action to "Set version as default"Delete these setup instructions fromREADME.mdwhen you are finished with them.