Extracting Tweets containing a particular Hashtag using Python - GeeksforGeeks

29 Dec, 2021
Twitter is one of the most popular social media platforms. The Twitter API provides the tools you need to contribute to, engage with, and analyze the conversation happening on Twitter, which finds a lot of application in fields like Data Analytics and Artificial Intelligence. This article focuses on how to extract tweets having a particular Hashtag starting from a given date.
Tweepy is a Python package meant for easy accessing of the Twitter API. Almost all the functionality provided by Twitter API can be used through Tweepy. To install this type the below command in the terminal.
pip install Tweepy
Pandas is a very powerful framework for data analysis in python. It is built on Numpy Package and its key data structure is a DataFrame where one can manipulate tabular data. To install this type the below command in the terminal.
pip install pandas
Create a Twitter Developer account and obtain your consumer secret key and access token
Install Tweepy and Pandas module on your system by running this command in Command Prompt
Import required modules.Create an explicit function to display tweet data.Create another function to scrape data regarding a given Hashtag using tweepy module.In the Driver Code assign Twitter Developer account credentials along with the Hashtag, initial date and number of tweets.Finally, call the function to scrape the data with Hashtag, initial date and number of tweets as argument.
Import required modules.
Create an explicit function to display tweet data.
Create another function to scrape data regarding a given Hashtag using tweepy module.
In the Driver Code assign Twitter Developer account credentials along with the Hashtag, initial date and number of tweets.
Finally, call the function to scrape the data with Hashtag, initial date and number of tweets as argument.
Below is the complete program based on the above approach:
Python
# Python Script to Extract tweets of a# particular Hashtag using Tweepy and Pandas # import modulesimport pandas as pdimport tweepy # function to display data of each tweetdef printtweetdata(n, ith_tweet):        print()        print(f"Tweet {n}:")        print(f"Username:{ith_tweet[0]}")        print(f"Description:{ith_tweet[1]}")        print(f"Location:{ith_tweet[2]}")        print(f"Following Count:{ith_tweet[3]}")        print(f"Follower Count:{ith_tweet[4]}")        print(f"Total Tweets:{ith_tweet[5]}")        print(f"Retweet Count:{ith_tweet[6]}")        print(f"Tweet Text:{ith_tweet[7]}")        print(f"Hashtags Used:{ith_tweet[8]}")  # function to perform data extractiondef scrape(words, date_since, numtweet):         # Creating DataFrame using pandas        db = pd.DataFrame(columns=['username',                                   'description',                                   'location',                                   'following',                                   'followers',                                   'totaltweets',                                   'retweetcount',                                   'text',                                   'hashtags'])         # We are using .Cursor() to search        # through twitter for the required tweets.        # The number of tweets can be        # restricted using .items(number of tweets)        tweets = tweepy.Cursor(api.search_tweets,                               words, lang="en",                               since_id=date_since,                               tweet_mode='extended').items(numtweet)          # .Cursor() returns an iterable object. Each item in        # the iterator has various attributes        # that you can access to        # get information about each tweet        list_tweets = [tweet for tweet in tweets]         # Counter to maintain Tweet Count        i = 1         # we will iterate over each tweet in the        # list for extracting information about each tweet        for tweet in list_tweets:                username = tweet.user.screen_name                description = tweet.user.description                location = tweet.user.location                following = tweet.user.friends_count                followers = tweet.user.followers_count                totaltweets = tweet.user.statuses_count                retweetcount = tweet.retweet_count                hashtags = tweet.entities['hashtags']                 # Retweets can be distinguished by                # a retweeted_status attribute,                # in case it is an invalid reference,                # except block will be executed                try:                        text = tweet.retweeted_status.full_text                except AttributeError:                        text = tweet.full_text                hashtext = list()                for j in range(0, len(hashtags)):                        hashtext.append(hashtags[j]['text'])                 # Here we are appending all the                # extracted information in the DataFrame                ith_tweet = [username, description,                             location, following,                             followers, totaltweets,                             retweetcount, text, hashtext]                db.loc[len(db)] = ith_tweet                 # Function call to print tweet data on screen                printtweetdata(i, ith_tweet)                i = i+1        filename = 'scraped_tweets.csv'         # we will save our database as a CSV file.        db.to_csv(filename) if __name__ == '__main__':         # Enter your own credentials obtained        # from your developer account        consumer_key = "XXXXXXXXXXXXXXXXXXXXX"        consumer_secret = "XXXXXXXXXXXXXXXXXXXXX"        access_key = "XXXXXXXXXXXXXXXXXXXXX"        access_secret = "XXXXXXXXXXXXXXXXXXXXX"          auth = tweepy.OAuthHandler(consumer_key, consumer_secret)        auth.set_access_token(access_key, access_secret)        api = tweepy.API(auth)         # Enter Hashtag and initial date        print("Enter Twitter HashTag to search for")        words = input()        print("Enter Date since The Tweets are required in yyyy-mm--dd")        date_since = input()         # number of tweets you want to extract in one run        numtweet = 100        scrape(words, date_since, numtweet)        print('Scraping has completed!')
Output:
Demo:
ParagPallavSingh1
Python-Tweepy
Python
Writing code in comment?
Please use ide.geeksforgeeks.org,
generate link and share the link here.
How to Install PIP on Windows ?
Check if element exists in list in Python
How To Convert Python Dictionary To JSON?
How to drop one or multiple columns in Pandas Dataframe
Python Classes and Objects
Python | Get unique values from a list
Python | os.path.join() method
Create a directory in Python
Defaultdict in Python
Python | Pandas dataframe.groupby()