oscardelben/rawler — pull request #35 by DanElbert (2014-05-09 18:49:51+00:00)
Title: Add ignore_fragments and fix query string handling

This pull request includes 2 changes:

1. I added an option to ignore URI fragments (I noticed that each link in my web app that had a fragment was counted as a separate URL)

2. Links that contained query strings would result in a request to the server without the query string. That has been fixed.