jmg/crawley — pull request #14 by oniltonmaciel (2014-07-11 19:59:19+00:00)
Title: Use urljoin to fix relative urls

https://docs.python.org/2/library/urlparse.html#urlparse.urljoin provides a robust way to make a relative url into a absolute one.

This fixes some issues like this one:

When accessing this url:
http://www1.abracom.org.br/cms/opencms/abracom/pt/associados

We find relative links like this:
`resultado_busca.html?letra=a`

The browser (chrome) build the absolute url like this:
http://www1.abracom.org.br/cms/opencms/abracom/pt/resultado_busca.html?letra=a

But crawley build the url like this:
http://www1.abracom.org.br/resultado_busca.html?letra=a

urljoin fix the issue, keeping the right behavior for `/relativeurl`:

In a hypothetical page http://mydomain.com/my/web/page.html:

'/relativeurl.html' link should become 'http://mydomain.com/relativeurl.html'

and

'relativeurl' link should become 'http://mydomain.com/my/web/relativeurl.html'