Extracting Data from Twitter using Python | by Shivani Dandir | Towards Data Science

Now that everything is being done online, the data shared on social media platforms has exponentially increased. Using this data we can analyze various socio-economic factors that are currently prevailing and a lot more. Here is a way to extract tweets based on certain topics from Twitter API using the Tweepy library.
For using the Twitter API you need to have a developer access Twitter account. Request for the same it might take 2–3 hours to get an approval. Once, you’re done with the set up create an app, in it, you will get Keys and tokens, which will help us retrieve data from Twitter. They act as login credentials.
Now, we will get on to code to get the tweets. First, you need to import all the packages required and initialize the token and key variables. OAuth essentially allows the user, via an authentication provider that they have previously successfully authenticated with, to give another website/service a limited access authentication token for authorization to additional resources.
#import files and give access to tokens and keysimport tweepy,jsonaccess_token=""access_token_secret=""consumer_key=""consumer_secret=""auth= tweepy.OAuthHandler(consumer_key,consumer_secret)auth.set_access_token(access_token,access_token_secret)
After getting access to Twitter data we’ll now create a file to save all the tweets in it.
tweet_list=[]class MyStreamListener(tweepy.StreamListener):    def __init__(self,api=None):        super(MyStreamListener,self).__init__()        self.num_tweets=0        self.file=open("tweet.txt","w")    def on_status(self,status):        tweet=status._json        self.file.write(json.dumps(tweet)+ '\n')        tweet_list.append(status)        self.num_tweets+=1        if self.num_tweets<1000:            return True        else:            return False        self.file.close()
We’ll now create a filter that will extract tweets based on certain words that are mentioned. Basically, it will extract tweets that contain the words which are valid for our project. For example, if you want data regarding covid19 you will use specific words like corona, coronavirus, covid19, etc to filter out the tweets as shown below.
#create streaming object and authenticatel = MyStreamListener()stream =tweepy.Stream(auth,l)#this line filters twiiter streams to capture data by keywordsstream.filter(track=['covid','corona','covid19','coronavirus','facemask','sanitizer','social-distancing'])
You can read the tweets stored in the file by following:-
tweets_data_path='copp.txt'tweets_data=[]tweets_file=open(tweets_data_path,"r")#read in tweets and store on listfor line in tweets_file:    tweet=json.loads(line)    tweets_data.append(tweet)tweets_file.close()print(tweets_data[0])
The output of the above is unrefined information with timestamps and certain ids that twitter might use. You need to clean the data to use it in your project. For cleaning the data there are steps to be followed such as removal of unwanted words, parsing of texts, removal of unwanted signs, Tokenisation, and a lot more.‘Cleaning the data’ that is a story for another time!
You can use the above format to get data for any of your projects be it election campaign, environmental issues, the stock market, or about lego. There are plenty of issues that can be analyzed using the power of Twitter. You just need to add related words or hashtags and voila you are good to deep dive into your project!
Please feel free to share your feedback, so that I can correct it if there’s a mistake. I am here to share my work and learn new things.
Here is the link for my Github profile.
Thank you! :)