Improved test coverage

In preparation for python 3 compatibility, these unit tests improve test coverage of some modules which use `urllib` functions.

`six` is added to requirements (I assume we are going to add it anyway). Version 1.5.2 is required because there was a [bug](https://bitbucket.org/gutworth/six/pull-request/22/moves-import-bug-fixed/diff) in `six.moves` in version 1.4.

Coverage is achieved:

    scrapy/contrib/downloadermiddleware/robotstxt.py  97%
    scrapy/contrib/linkextractors/htmlparser.py       96%
    scrapy/contrib/linkextractors/regex.py            100%

In next PR, I plan to replace various `urllib` exports with `six.moves.urllib`.