Web Scraping in a corporate environment — The Scraper
By: Arno Schots
Tags: Web Scraping, Python, Oracle, Data Science

Photo by Stephen Dawson on Unsplash

Project — The Scraper

I decided to to bundle my efforts, at least the tangible code part, under the code name The Scraper. Working for an engineering company, I have this natural urge to automate & engineer my own work processes.

Let’s talk about my project. Basically I have a web based reporting system that doesn’t allow exports/downloads nor allow me to operationalise and execute on the data. In this case it is not a feature issue, but it is how the system is designed and setup based on corporate guidelines. The reporting system is Oracle Business Intelligence 12 Enterprise Edition (EE).

Now, the good thing is, the reports are basically dynamically generated HTML pages — including HTML tables, so I started thinking about writing a web page scraper — to scrape the data from the webpage and store it somewhere else. Keep in mind that I just want the data from the displayed table!

What better language to use then Python! Python has gained huge popularity over the last couple of years, not in the last place because of it’s simplicity and huge eco system of modules. Also it is a very popular language for data science purposes. For this exercise I am using Python 3.7.

Enter “Requests” & “BeautifulSoup” Python Modules

Python has an excellent module called “Requests” that allows to make calls to webpages. This is article is not going to talk in depth about these technologies as there are plenty of articles out there. I want to share how I applied the technologies in a real life context and some of the specific challenges I encountered.

Let’s start coding…note that you need to have the requests module installed, wich can be done with:

pip install requests

After the module is installed we can fire up our code editor and start with importing the requests module:

import requests

Python Modules Import in the source code of the program

As you can see I also import “getpass”, “time” and “BeautifulSoup”. We will need these later on, so again use pip install [module name] to install them on your system.

BeautifulSoup is one of the most frequently used HTML parsers. It converts HTML in a python nested object. This allows for easy navigation through HTML and finding specific parts in the HTML code based on HTML TAGs. We will use it later, because we first need to get the page to be parsed.

Calling the application page

As you could have guessed, we will use requests to call the page we want. Let’s imagine the base URL is https://bi.yourcompany.com — but off course this will only get you to the homepage. Not practical. The good thing about Oracle BI EE is the fact that you can create the page view on the data that you are interested in and create a bookmark link that will remember your data selection options and hence, you only have to call this bookmarked URL to get the data you want. This will normally be a long string like (note: I shortened the string a lot where you see the “….”:

https://bi.yourcompany.com /analytics/saw.dllPortalGo&PortalPath=%2Fshared%2FSales%20Forecast%20and%20Pipeline.......ghvgo90negq&options=bsmrdle

But before getting the URL, we need to do two more things:

Setting the header so we appear to be a normal user instead of a programmatic interface to the server — some web-servers block scraper based access Setting up a “session” so we can store information (e.g. cookies) between requests as we have to deal with authentication

So we can set the header like the following:

headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux i686; rv:7.0.1) Gecko/20100101 Firefox/7.0.1'}

In this case we are using a fairly common header, used by the Firefox browser, to communicate with the server.

Now that we have done this, we will create a session object, which we can do like this:

s = requests.session()

Next we will update the headers values based on this newly created session object:

s.headers.update(headers)

Now we call the full URL for the first time:

r = s.get('https://bi.yourcompany.com /analytics/saw.dllPortalGo&PortalPath=%2Fshared%2FSales%20Forecast%20and%20Pipeline.......ghvgo90negq&options=bsmrdle')

In this case, if you have read the requests documentation, you know that “r” will be the response object and, after a successful request, will contain all the response data from the server, including the raw HTML we need.

You can inspect the content by doing this:

print(r.text)

There are much more things you can do with the response object, but this is out of scope of this post.

Getting the data from the response and write it to file

Finally we get to the end of our exercise. At this moment it is enough to just store the HTML so we can process it later with BeautifulSoup. Writing the data is pretty straightforward.

The code to do it is:

with open('output.html', 'w') as f:

f.write(r.text)

What we do is open/create/close a file with the name “output.html” and as we want to “write” to the file, we use the “w” flag. The object name that we will use to interact with the file is “f”. Then we simply call the write operation on the “f” object and use “r.text” data to write to the file — et voila! Double click on the “output.html” file to open it in your browser and see your locally stored data.

What’s next

I can hear you thinking, but what about the other modules you mentioned like “time”, “getpass” and “BeautifulSoup”? We will deal with those in subsequent posts. Also, you didn’t think it was this easy right to get the data? Right. I made this example simple, for the sake of the exercise. We have to deal with authentication, HTML parsing, waiting for pages to load, error handling etc. Stay tuned for more…and thanks for reading. Leave comments, questions and feedback if you want.