How to Fetch Data from the Twitter API Using Twurl and Ruby
By: Ethan Fournier
Tags: Coding, Flatiron School, Ruby, API, Twitter

For my latest project at Flatiron School, I was tasked to make a single-page website with a JavaScript front-end and a Ruby on Rails back-end. Our back-end would work as our own API that we would fetch from in our front-end.

My project idea was to create a dog rating app. I was inspired by WeRateDogs® on Twitter and wanted to recreate it in a website. I wanted to pull the images from the Twitter feed and then seed my database with these images in the back-end.

So, how do you go about doing that? I started searching for how to scrape Twitter, and I found Twitter’s official website for its API. There are a few steps before you can actually have access to this API. The first step is to apply for a developer account: https://developer.twitter.com/en/apply-for-access.

If you don’t have a Twitter account already, sign up for one. Afterwards, you have to explain to Twitter why you need access. You have to choose what your purpose is (I chose Academic: Student), describe your intended use and then agree to their terms. I got a follow-up email asking for more details on my project, and then I was approved the same day.

Once you’re approved, you have to create an app. This is pretty simple. Decide a name, describe it, and give it a website URL. You can use a Heroku URL if you’re making your own application and don’t have a URL yet.

Okay, I have access to Twitter’s API. Now what? I started searching for documentation on how to use the API. What are the easiest ways to access the data? There are several different methods of making fetch requests to the API; you can use JavaScript, Ruby, Python, Curl, and Twitter’s own version of Curl, Twurl. Curl is a command line tool that can be used to make various requests to the Internet.

After playing around with a few different ways of requesting the data using Postman (an application that can make fetch requests for you), I decided on using Twurl. Twurl takes the complexity of scraping data from an API out of your application and simplifies it for you in the command line.

How do you add Twurl to your Ruby application? Go to your Gemfile and add gem 'twurl' . You will then have to authorize your app. Go to your app on the Twitter Developer website and click “Keys and Tokens”. Then go to your terminal and enter the following command:

The second key needs to be replaced with the API key found under the Consumer API keys header and the second secret needs to be replaced with the API secret key.

The command line will then prompt you to go to a URL. Copy that URL in your browser, click to authorize the app, and then copy and paste the number they give you back into your command line.

You’re good to go. Now, let’s make your first GET request. First, decide which Twitter feed you want to scrape. I tried scraping WeRateDogs, but alas, they have copyright protection and would not give me access to their data, so I tried the next best feed: CuteEmergency.

Type the following in your command line, changing the value after screen_name= for the screen name of the twitter feed you want to scrape:

The count is how many tweets you want to get back. Two hundred is the maximum amount of tweets that Twitter will allow you to access at a time.

Type enter and whoa! Your terminal is filled with JSON! Okay, cool, I have my data. Now what? What can I do with this giant ugly JSON object in my terminal?

Run the same command line again (use the up arrow key to get it back) and then add one addendum to it > cuteemergency.json (or whatever you want to call that file).

The whole line looks like this:

If you look in your app folder, you will see the file name cuteemergency.json . If you’re using a text editor like Atom or VS Code, it will not be able to be formatted properly, as the file will be too big for the editor to parse. Sublime will format it with an extension, Pretty JSON. To look at the content formatted in the browser, you can go to https://linangdata.com/json-formatter/ and paste the JSON data in there.

Okay, great, you have a lot of JSON data. What now? How can I actually use it in my app?

First, go to your seeds file and write this line:

This will give you a nicely parsed JSON object to work with.

Next, you need to find out what data you want to access. Look through your formatted data and find the specific attributes you want to work with. I wanted the pictures from each tweet. I looked through the JSON and found what I needed: “media_url”. But wait, this is a nested object. How do I pull out the specific data I want in a nested JSON object?

Through a lot of searching and going through a multitude of Stack Overflow questions, I stumbled upon another gem, Hashie. To use it, add this to your Gemfile: gem 'hashie' . Wherever you want to use this gem, you’ll have to extend it: [object].extend Hashie::Extensions::DeepFind . You will then have to use the [object].deep_find("[key you are looking for"]) command to access that piece of data.

I used it in the following function:

What this gem does is look deep within the hash (or object) and find the key you are looking for. This gem looks at the deepest item first, so keep that in mind.

This newDogs function now returns only the tweets that have a media_url . Now I can use newDogs to seed my database:

And now you have a database filled with pictures of dogs!

You can then fetch the dogs from your database with a JavaScript fetch function on your front-end:

Now you too can get whatever data you need from Twitter.