Generate a corpus of public “tweets” containing a subject/keyword of interest | by Daniel Ellis | Towards Data Science

It is often hard to conduct research into peoples interest, with poll and survey results often being skewed, or there being unintentionally loaded questions. It, therefore, means that the ‘truest’ method of understanding peoples interests lies in what they wish to share or talk about without any external influences. The simplest way to do this is by scraping the web for twitter posts containing certain characteristics.
Initially, the most intuitive method to do this would be to look at the twitter API. To use this you first have to register for a developer account (free) at https://developer.twitter.com/en/apply-for-access
Here you are assigned a set of credentials, which you can save as a JSON file in the following format:
{ “twitter”: { “auth”: {     “consumer_key”: “xxxx”,     “consumer_secret”: “xxx”,     “access_token_key”: “xxx”,     “access_token_secret”: “xxx” },  “token”: “xxx”,  “secret”: “xxx” }}
Next, we need to install the Twython library pip install twython(python 3 unfortunately), after which we can customise the following script to our needs.
# Import the Twython classfrom twython import Twythonimport json# Load credentials from json filewith open("twitter_credentials.json", "r") as file:    creds = json.load(file)['twitter']['auth']# Instantiate an objectpython_tweets = Twython(creds['consumer_key'], creds['consumer_secret'])# Create our queryquery = {'q': 'air quality',        'result_type': 'popular',  # other options 'mixed'        'count': 100,   # max 100         # 'until':"2019-02-01",        }import pandas as pd# Search tweetsdict_ = {'user': [], 'date': [], 'text': [], 'favorite_count': []}for status in python_tweets.search(**query)['statuses']:    dict_['user'].append(status['user']['screen_name'])    dict_['date'].append(status['created_at'])    dict_['text'].append(status['text'])    dict_['favorite_count'].append(status['favorite_count'])# Structure data in a pandas DataFrame for easier manipulationdf = pd.DataFrame(dict_)
The downside to this method is that results are limited to the last 7 days, and can only return up to 100 tweets.
Instead, if we wish to get all the tweets between dates further into the past, we need to rely on other methods. To do this we can rely on the GetTweetsFrom library which is available at; https://github.com/Jefferson-Henrique/GetOldTweets-python
This is designed on python 2, although has an experimental interface to python 3 which may or may not work.
As I wish to only extract the text of each tweet and save it on a new line for text processing (word2vec), the original example code supplied with the library was tweaked slightly. Here I have hard-coded the query, start and end dates, and an output filename.
The script below cycles through each day between the two supplied dates and downloads the first 1000 tweets it finds matching the query. This can be adapted to the user’s needs.
# -*- coding: utf-8 -*-import sys,getopt,datetime,codecsfrom datetime import date, timedeltaedate = date(2019, 10, 1)   # start datesdate = date(2018, 10, 1)   # end datequery = 'atmospheric chemistry'maxtweets = 1000toptweet=FalseoutputFileName = 'myoutput.csv'delta = edate - sdate       # as timedeltadates = [(sdate + timedelta(days=i)) for i in range(delta.days + 1)]if sys.version_info[0] < 3:    import gotelse:    import got3 as gotdef main():try:tweetCriteria = got.manager.TweetCriteria()tweetCriteria.querySearch = query        tweetCriteria.topTweets = toptweet        tweetCriteria.maxTweets = maxtweetsoutputFile = codecs.open(outputFileName, "w+", "utf-8")def receiveBuffer(tweets):            for t in tweets:                outputFile.write('\n%s'% t.text)            outputFile.flush()for dt in dates:            print (dt)            tweetCriteria.since = str(dt)            tweetCriteria.until = str(dt + timedelta(days=1))got.manager.TweetManager.getTweets(tweetCriteria, receiveBuffer)except arg:        print('Arguments parser error, try -h' + arg)   finally:        outputFile.close()        print('Done. Output file generated "%s".' % outputFileName)if __name__ == '__main__':   main()
The Twitter API works only for collecting tweets in the last 7 days.
The GetOldTweets library is useful if you need historic data
The output file generated can then be used as a corpus for NLP based machine learning and prediction, or sampling of the public views on the subject.
Both methods are pretty easy to use and are almost plug-in and play (once you get the correct version of python working).