Twitter bot for Data Scientists. Ever wonder how to create your... | by Stephen Oni | Towards Data Science

Ever wonder how to create your @Getvideobot @threadreaderapp and the likes
Twitter is one of the most widely used social platforms. It’s used by individuals and organizations. It serves as a large pool of information for those whose information (data) drives their lives and work.
The existence of twitter API has provided unlimited creative access to the information shared and provided on the platform, due to this, different twitter bot has being created for a different purpose, and it has helped make twitter more of a better platform.
Going further than just making twitter platform interesting with the use of these bots, it can serve as a means to crowdsource data for data scientists.
In this article we will be learning how to create different twitter bot to crowdsource data for your data science project, like how to:
download images and videos by monitoring specific keywords.download media data by being referenced in a tweet.collecting tweet of a monitored user.Unrolling twitter thread in to a text file.storing data gather in aws s3.
download images and videos by monitoring specific keywords.
download media data by being referenced in a tweet.
collecting tweet of a monitored user.
Unrolling twitter thread in to a text file.
storing data gather in aws s3.
These various processes can help crowdsource data not just for NLP, but also for computer vision( since most people think twitter is most best for collecting data for Natural language processing).
In this article, I will assume you have Twitter API Authentication Credentials and also know how to use Tweepy. If not, check the reference below on how to obtain twitter API credentials and how to install tweepy and gain a little bit of introduction to it. But I will advise you follow through the article first, just to gain the application view, then go back to the reference.
Provided you have the twitter API credentials, the first thing to do is to create a config file, which sets up your authentication with twitter API.
A call to the function helps checks if the authentication is valid. You can replace the os.getenv() with the api keys directly, or you store the api key in your environment . The wait_on_rate_limit and wait_on_rate_limit_notify helps tweepy wait and print out a message when the rate limit is exceeded.
Download Media Data by Monitoring Specific Keywords
Data collection is the backbone of every data-driven project. The type of data being collected depends on the type of project being done. For example, for a computer vision project, it requires images or videos. Twitter can serve as a platform to curate such data in different ways. And one of the ways is monitoring a particular set of keywords related to the type of data you want for your project.
Let say you want to curate or gather data, involving Trump, a twitter bot can be created to always listen to every tweet mentioning Trump.
The Monitor bot is created using Twitter Stream service api interfaced by tweepy. This helps the bot to keep listen to different tweets across the Twitter platform. The Monitor class inherit from the tweepy.StreamListener and it contains two methods; on_status this helps to pass in the tweet and do whatever we want with the tweet passed in and on_error provide the actionable point in case an error is being encountered during the on_status process.
The tweet passed in on_status contains the JSON attribute of each tweet. Since in this example we need the media data. The json element of the tweet contains what will be called entities and the entities contain the media files and the embedded url in a tweet. For example
{u'hashtags': [{u'indices': [0, 12], u'text': u'NeurIPS2019'}], u'media': [{u'display_url': u'pic.twitter.com/GjpvEte86u',   u'expanded_url': u'https://twitter.com/doomie/status/1206025007261876226/photo/1',   u'features': {u'large': {u'faces': []},    u'medium': {u'faces': []},    u'orig': {u'faces': []},    u'small': {u'faces': []}},   u'id': 1206024997262618626,   u'id_str': u'1206024997262618626',   u'indices': [278, 301],   u'media_url': u'http://pbs.twimg.com/media/ELypWGAUcAIpNTu.jpg',   u'media_url_https': u'https://pbs.twimg.com/media/ELypWGAUcAIpNTu.jpg',   u'sizes': {u'large': {u'h': 1536, u'resize': u'fit', u'w': 2048},    u'medium': {u'h': 900, u'resize': u'fit', u'w': 1200},    u'small': {u'h': 510, u'resize': u'fit', u'w': 680},    u'thumb': {u'h': 150, u'resize': u'crop', u'w': 150}},   u'type': u'photo',   u'url': u'https://t.co/GjpvEte86u'}], u'symbols': [], u'urls': [], u'user_mentions': []}
the tweet.entities contains this keys and value items. since we will need the media files. For the image file, the media_url will be used to download the image in a tweet. and for the videos uploaded in the tweet will be downloaded using the expanded_url and for the videos embedded through a url, such as youtube videos. will be extracted from the urls , but in this tweet, it contains no urls, that's why the array is empty. But for a tweet which contains a url
{ ....  u'urls': [{...,u'expanded_url':'youtube.com?...}]
Hence the video is downloaded by parsing in urls[0]['expanded_url'] .
Various python libraries or modules have been created to make the downloading of images and videos easy. To download images, a python package called wget is used. For youtube video download pytube is used and for twitter videos, twitter_dl is used. All of these libraries are great libraries, they make it easier to build this project. But for twitter-dl and pytube i make some changes to them due to the error I was encountering, but i think the new update should resolve that. So you can check my github to download them if you encounter any error.
To download these media files, we combine these libraries into a single file
Now we can combine this with the Monitor.py to start downloading media files
Download Media file When being Referenced to a Tweet
Sometimes instead of trying to monitor a specific keyword, we can let others help crowdsource data for us on twitter. The method involves letting people reference your twitter name under the tweet that contains the kind of data you want. The process is similar to that of the monitoring process just that we won’t be using streaming api, and it does not listen to all tweets except the one you are being mentioned in.
The mention function makes use of the api.mention , to fetch all the tweet you are being mentioned in. To prevent the api.Cursor from fetching all the tweets you are being mentioned in, since the time you join twitter, we use the since_id . To make sure it keeps fetching the newly tweet you are being mentioned in. First, we get the id of the tweet of your last mention.
tweety = []for tweet in tweepy.Cursor(api.mentions_timeline).items():    if len(tweety) > 1:        break    else:        tweety.append(tweet)
And since the tweet is being fetched based on the newly created tweet. we only need that and then we index tweety to get the id of the tweet.
tweety[0].idoutput[]:1208298926627074049
And in mention() in the above code palette, we make use of max(tweet.id,new_since_id) This is used since for each new tweet, the id is always greater than the tweet id before it. that is new tweet id is higher than old tweet id.
Don’t forget that the tweet collected are not just text but json element, containing tweet attribute. For this bot, you are being referenced to a tweet, via a comment section. The main content you want is in the tweet the user reference you in via a comment. So the user is replying to a tweet.
In the tweet json element, it is possible to extract the id of the tweet the user replied to. We call this id in_reply_to_status_id. When a user is not replying to a tweet, the id is usually None . To create the bot, we first check if the reply status id is not None, then we process the tweet. Then we get the tweet from the status id using
status_id = tweet.in_reply_to_status_id                                   tweet_u = api.get_status(status_id,tweet_mode='extended')
Then you can now print out the text contained within the tweet.
After this basic process, the remaining process is similar to that of the Monitor bot, if the media files are required. For the code above we can just store the text in a database. And such a process will still be required to create another bot called popularly know has Unrollthread which will be discussed before the end of this article.
To download media data
Collecting tweet of a Monitored User
A Twitter bot can be created to always track twitter feeds from a particular set of users, this can help curate data to create a language model base on a specific data and for various NLP data analysis. In retrospect to the Monitor specific keyword bot created at first, but for this process instead of subscribing to a keyword we subscribe to a user using their user id.
In the code, I specify the user id, the user id can be retrieved by; e.g
api.get_user("@_mytwtbot").id #specify the user screen name you want//1138739106031308800
Hence, you will always be the first to get the tweet created by this user.
Unrolling Twitter thread into a text file.
You’ve heard the unrolled thread bot on twitter, to always create a pdf format of a thread. We will be creating that. that can also help gather data for your data science project. The process is still the same with the Mention bot. It makes use of the reply status id, and it uses a recursive function to get all the tweets in a thread.
Storing Data gathered in AWS s3
Now that we’ve have different approaches to collect data, we need to be able to store them. Aws s3 provides one of the quick and easy ways of storing media files online if you don’t have a storage services online.
First, you install aws sdk, which is provided via boto3 a python library
pip install boto3
Then for boto3 to work and connect to your aws resources, you need aws credentials which can be obtained by creating an iam user permission, which then can be stored, in your computer home directory
touch ~/.aws/credentials
Then in the credentials, we store the necessary info
[default]aws_access_key_id = YOUR_ACCESS_KEY_IDaws_secret_access_key = YOUR_SECRET_ACCESS_KEY
Now to obtain the access keys;
First, visit the aws console and enter the iam resources
Follow the pictures in order of how it is arranged from left to right to set up an iam on aws. Once this is done you can download the csv as seen in the last image above, copy the credentials in it and paste in the ~/.aws/credentials
Once this is done we go ahead to create a bucket in aws S3.
Click on create bucket to create a bucket to store your files.
Then specify the name, the region you want and then click on create hence your bucket is created.
Since the bucket is created, we can always make a call to it using boto3:
import boto3s3 = boto3.client("s3")s3.upload_file(path,"bucket_name","name_to_store_file")
This the basic function we will be using to store the files on s3. the upload_file method takes in the path of the file to be uploaded e.g /home/documents/bot/video.mp4 and the bucket_name is name of the bucket you created, then the third argument is the name you want to store the file as on s3.
So with a little bit of modification to our download_t.py we can save the file on aws s3
After the files are being uploaded it can be viewed by clicking on the bucket
Now everything is set, we can now create a docker image and then deploy to aws. Check the reference below to learn how to deploy to aws.
But just one little addition after you’ve learnt how to deploy to aws from the reference link, you add the aws credentials alongside with the environmental variable
The additional thing to add to this, in order to be able to save the media files to aws s3
-e aws_access_key_id="sjbkjgkjhfyfyjfyjyfj" \-e aws_secret_access_key="1124knlkhk45nklhl" \fav-retweet-bot
Now as a data scientist we’ve learned a way to gather data from twitter and create a bot. Check out this to learn how to create a twitter bot using deep learning.
Download the repo containing the code from github
REFERENCE
This post heavily depends on the work of some team from realpython.com check here to learn how to generate twitter api and how to deploy to aws .