Package: twitter-toolkit
Twitter toolsThis project contains different tools to help search and analyze in twitter.The analysis of tweets is often followed by doing the same pipeline for different projects.
This toolkit is a compilation and wrapper of many tools to ease the pipeline of analysis in twitter.
First of all it provides search utilities, either by searching by a twitter query or by identifier.
Then it integrates some models to infer users age, gender, and if it is a person or an organisation. This inference should be used with caution since it is not perfect, but can yield an overview of the type of users analyzed.
There is also a location of users inference for Spanish locations based on theirlocationtext ordescriptionin their Twitter profile.
For the text analysis we provide a pipeline for Topic analysis using the LDA algorithm and some sentiment analysis too.
Finally we provide a network creation of the tweets and users function for a network analysis.Twitter SearchCredentialsTo run this you need to provide your Twitter API credentials in the form of
YAML file.For example:search_tweets_api:endpoint:https://api.twitter.com/2/tweets/search/allconsumer_key:XXXXXXXXXXXXconsumer_secret:XXXXXXXXXXXXXXXXbearer_token:XXXXXXXXXXXXXXXXXSearching tweetsYou query tweets withsearch_tweets_by_query.To have a more detailed for the parameters take a look at theTwitter APIfromtwitter_tools.search_toolsimportTweetSearchUtiltsu=TweetSearchUtil('path_to_yaml_creds')tweets=tsu.search_tweets_by_query("python OR #python",tweet_fields="author_id,conversation_id,created_at,id,in_reply_to_user_id,lang,public_metrics,text")Searching by idYou can also search retrieve tweets and users by their id.fromtwitter_tools.search_toolsimportTweetSearchUtiltsu=TweetSearchUtil('path_to_yaml_creds')tweets=tsu.retreive_tweets_by_id(['12341','12342'],tweet_fields="author_id,conversation_id,created_at,id,in_reply_to_user_id,lang,public_metrics,text")users=tsu.retreive_users_by_id(['4321','4322'],user_fields="created_at,description,id,name,profile_image_url,public_metrics,username")Twitter InferenceThis is a Wrapper ofM3Inferencebut with an ease to use and make a general pipeline with this set of tools.fromtwitter_tools.user_inferenceimportTwitterUserInferenceusers=[{...},...]tui=TwitterUserInference()inference=tui.infer_users(users,lang='en')Users LocationThis tool is only available for Spain locations.To feature other countries, a json in the format asplaces_spain.jsonshould
be added.This tools checks the location of an user based otheir text location and
description when no geolocation is available.
Checks for city/country/region words in the user profile to try to identify for
its location.fromlocation.location_detectorimportLocationDetectoruser={...}detector=LocationDetector('path_to_places_json')loc,method=detector.identify_location(user['location'],user['description'])Topic analysisThis tool will do every step of topic analysis using LDA.The typical pipeline can be represented by as follows.fromtwitter_tools.topic_analysisimportTopicAnalysistweets=[...]analyzer=TopicAnalysis(language='es')tweets_clean=analyser.clean_docs(tweets)tweets_lemmas=analyser.lemmatize(tweets_clean,filter_postags=['ADJ','ADV','NOUN','VERB'])ldamodel,docs_dict=analyzer.topic_analysis(tweets_lemmas,topics_nb=10,print_words=10)Sentiment AnalysisSentiment analysis of text using pretrained models.fromtwitter_tools.topic_analysisimportTopicAnalysistweets=[...]analyzer=TopicAnalysis(language='es')sentiments=[analyzer.sentiment_analysis(t)fortintweets]Network creationThis tool creates graphs based on the tweets and users interactions.It can create the user and the tweet graph.The tweets dict like object must contain at least the following fields:id,retweeted_by,favorited_by.The users dict like object must contain at least the following fields:id,screen_name.fromtwitter_tools.network_toolsimportcreate_tweets_network,create_users_networkusers=[...]tweets=[...]T=create_tweets_network(tweets)U=create_users_network(users,tweets)Once the network create you can export it and open the file in Gephi to visualize it and analize it.importnetworkxasnxnx.write_gml(T,"tweets_network.gml")