busbud/coding-challenge-crawler-b — pull request #1 by Lazik (2014-02-13 20:22:12+00:00)
Title: Loïc Maltais-Herry pull request

Solution uses mechanize with beautifulsoup

Use mechanize to handle the requests.
Use custom functions to extract ajax response information & manipulate form data.

As per the requirements, did not use selenium as it requires a driver.
Solution has a very low memory footprint ~20 MB. (As opposed to selenium, 100MB per instance)

Solution is not multi-threaded but can be easily extended.

Solution is production ready, handles all special cases and malformed output from their webapp.
See .json output & more info in note.txt.

More details:
The website use an apsx app
Two main challenges:
1 - Non-uniform + malformed station location data

2 - AJAX steps coupled to the aspx app + JS event handler
To get the bus routes and bus stop info
# Step 1 : Origin AJAX #1
Click on the drop down menu to select your origin
Browser sends ajax request to get the destination data.
The response gets a new view_state & event_validation to make a valid aspx request

# Step 2 : Select Destination AJAX #2
At this point you need to request a valid "Destination Selected" event_validation from the aspx app. (Even if you have the route data for Step 3, the aspx app won't give you the data because of the apsx state)
Forge post_data, make the post request and extract new aspx states from the response.

# Step 3
Forge the post_data with the step2 aspx states and sent it. Profit.