The Complete Data Mining Pipeline
By: Pablo T. Campos
Tags: Python, Data Science, Postgresql, Selenium, Programming

Now that we know how to obtain xpaths, let’s write some code! We’re going to create a login function that receives two parameters, username and password, find their respective html input fields and write them in. For this we use find_element_by_xpath() to obtain the web element we want and send_keys() to write some text in them.

Good code should be elegant and full xpaths are certainly not elegant. For this reason, I chose to craft some more refined xpaths to find username_input , pswd_input , and login . You can use the xpaths you copied from the browser — they will work just fine — but don’t be scared to use mine and try to understand how they work!

Let’s cover what they do by analyzing the xpath to username_input . First of all, // selects any element in the document that matches the following description. In this case, we’re looking for an input element that has an attribute called name equal to session[username_or_email] . In essence, it achieves the same thing as the full xpath we can copy from the page, but this way looks nicer.

Notice that at the beginning of the method there’s a sleep() function. This will halt the execution of our scraper for the amount of seconds we specify. This is extremely useful as we give time for the pages to load before we try to access any of their elements.

If you’re trying to access an existing element of the page and you get an error like the following, consider increasing the amount of time you want to wait for the page:

selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element:[...]

Increasing the amount of time you want to wait for the page gives it a chance to fully load before we perform an operation.

Now that we have our __login function inside the TBot , lets call it from the constructor. It should look something like this:

I strongly recommend that you store your username and your password in another Python file. This way you can import your username and password into the main.py function and use them without showing their actual values. Let’s see if it worked — run your main.py with the following command and you should get to your Twitter home page automatically:

(venv)$ python main.py

Extracting our much-desired data

Now comes the juicy part, extracting all the relevant information from our Twitter feed. First, we need to make a plan for how to proceed. Take your time, examine the HTML of the page, and come up with the best approach for achieving your goal.

Based on my experience, probably all tweets have the same HTML structure (true in this case — always trust your experience). This allows us to write code to get information out of one tweet and reuse it for the entirety of our twitter feed. Let’s get to it!

Inside the TBot class create the following method:

Notice how we have stored the output of find_elements_by_xpath in tweets , which means that we have managed to capture all tweets under one single xpath. Now we iterate through each tweet, extracting its individual information. Each tweet, in turn, possesses the same internal HTML structure, so we can use the same functions to extract information.

For this approach, we can’t just copy and paste the xpath we copied from the browser, as each tweet will have its own xpath. We need to be able to generalize our way to locate tweets inside the document, so need to craft our own xpath.

But I’m not really satisfied with this solution. For comments, retweets, and likes it doesn’t obtain the actual number, just the string that represents it. For example, 1.9K is a string while 1900 is an actual number we can work this. To fix this problem I created the following method:

This method checks if data contained either a K for thousands or an M for millions, then removes it, turns data into a decimal number, and multiplies it by its respective amount. Because it’s not nice to see someone got 1900.0 retweets, it finally turns data into an integer.

Implementing this method inside __scrap_tweets is fairly simple, we just need to call it for the values of comments, retweets, and likes and capture the output value:

Well, all that’s left to do is call the method from the constructor, which should now look like this:

Let's check if it works with (venv)$ python main.py . You should now see your whole Twitter feed followed by this nice error message:

selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document

If you happened to somehow avoid this error, you should still listen closely to this. When we load all the tweets from our Twitter feed into tweets , the last of them aren’t actually being displayed on the page. So, when we try to access their data, Python gets confused and stops its execution. To avoid this problem, we need to scroll down each time we get an error so that new tweets are loaded into the page. Let’s create a scroll method:

Now we need to wrap the code inside __scrap_tweets in a try/except and call the __scroll method in the except section. It should end up looking something like this:

But we don’t want to only obtain data from our initial feed when we can scroll down indefinitely. We’re going to call __scrap_tweets periodically from our infinite loop at TBot constructor, resulting in something like this:

And when you run (venv)$ python main.py your terminal should be flooded with endless tweets from your feed.

But we don’t want to store them in the output log of our terminal do we? We want to store them somewhere we can extract them when we need them: in a database.