busbud/coding-challenge-crawler-b — pull request #4 by scjody (2014-04-14 16:20:46+00:00)
Title: Working Megabus crawler by Jody McIntyre

Here is my submission to the [Busbud crawler coding challenge](https://github.com/busbud/coding-challenge-crawler-b).

# Overall approach

I chose to use the [Scrapy](http://scrapy.org/) framework because it's a widely-used and powerful framework specifically designed for scraping data from web pages. The main alternative I considered was writing my own parser based on an HTML library like [Beautiful Soup](http://www.crummy.com/software/BeautifulSoup/) and writing my own event loop, but Scrapy has a lot of functionality built in so I believe this approach was faster to develop while still being maintainable (the learning curve for Scrapy is pretty small and its approaches to things are very Pythonic). Also since Scrapy is widely-used, the event loop has been well tested and is likely to keep working as the web changes.

The one downside I can see with Scrapy is that it forces the use of Python 2.7 rather than 3.x, but your requirements specifically asked for Python 2.7 (and Python 2.7 support has now been extended to 2020) so I think this is fine.

## Scheduling

One of the requirements was to produce code that could easily be run as part of a scheduled process. My solution allows at least 3 ways to do this:

1. The script could, of course, be run from cron (or a similar scheduler) using run.py.
2. scraper.py was designed to be required by other Python code, so it could be integrated into a Python-based scheduling system.
3. Scrapy itself has a scheduling facility called [Scrapyd](http://scrapyd.readthedocs.org/en/latest/). This code has not been tested with scrapyd, but I expect it would work with minimal (if any) changes.

# Goals and tradeoffs

## Maintainability

Software should of course be maintainable by the original developer and by others. Scraper code is always at least somewhat "hairy", but I tried to minimize this:

* CSS-based selectors are used to find data whenever possible, rather than XPath-based selectors. When XPath was necessary, the minimum was used to keep it simple for programmers who are unfamiliar with XPath.
* Regular expressions were used only when they shortened or simplified the code significantly.
* Repeated code is kept to a minimum using a small class structure (common spider code is in MegabusSpider, which is the parent class for other spiders), and by creating helper functions.
* The code is pep8 clean apart from the option parsing that was part of the initial run.py (I'd be happy to clean this up, but I'm not sure how your developer culture feels about that sort of change).
* The code is pyflakes clean, and mostly pylint clean. The pylint items that remain are matters of taste (helper functions as class members when they could be outside the class, for example) and I'd be happy to do things differently if your company's practices suggest to.

## Resilince

Scrapers are, by their nature, fragile. I have taken a few steps to address this:

* Every POST is based on a previous response, so if extra fields that don't affect the data being requested (such as CSRF keys) are added to a form, the scraper will continue to work.
* The code is designed to raise exceptions when data is missing or in an unexpected format, rather than producing bad data.

In production, additional steps could be taken, such as automatically comparing data to previous runs and flagging major differences for manual review. (For example: we've been receiving 32 stops normally, but now we only get 12, or we get 500).

# Notes on specific choices and steps

## General

* I assumed that we want to create a new JSON file every time, so the output file is removed if it exists. This would be trivial to change.
* Scrapy is somewhat noisy by default - a summary is produced at the end of each run, and for longer runs, status is printed. This is easy to turn off by changing the loglevel passed to log.start in scraper.py.

## Step 1 (stops)

* stop_location is left blank in the output when none is returned by Megabus. If the software that uses stops.json absolutely needs a stop_location, we could set it to stop_name, but I feel it's best to leave it blank so different users of stops.json can do whatever's appropriate to their needs.
* stop_location isn't great in general. The original data seems to be entered by a human without reference to other stop locations. I did a bit of cleanup with regular expressions, but some are still fairly bad, and capitalization is often wrong (e.g. "Giant tiger" instead of "Giant Tiger" or "Mcmaster" instead of "McMaster". The best way to produce perfect stop_locations is probably to manually rewrite every one, and flag unknown stop_locations for rewriting. Whether or not this is worth the time is a business decision :)

## Step 2 (routes)

* It would be more efficient to collect stops and routes together, since all the requests needed to collect routes are already made when collecting stops.

## Step 3 (departures)

* http://ca.megabus.com/Default.aspx uses JourneyResults.aspx to return departure data, which is in robots.txt. If Busbud wants to run this crawler (or something similar) regularly, it should get permission from Megabus, and ideally an exception in robots.txt.

* Timezones are not specificed anywhere. It appears that Megabus uses the local time of the bus stop, so that's what the scraper returns. In any case this is fine for now since every Megabus Canada destination uses Eastern time, but it could be a consideration for the future.

* I added a command line argument for the location of the routes.json file. It defaults to looking in the current directory.