Merge pull request #62 from andresp99999/fix-content-type

Fix exception when Content-Type is empty and response type is not Html
Fix small typo
Merge pull request #63 from amferraz/patch-1

Fix small typo
pass explicit encoding in url processors. Added testcase which motivated the fix.
Merge pull request #64 from kalessin/encoding

pass explicit encoding in url processors. Added testcase which motivated the fix.
Add support to define allowed_domains inside the spider definition
Add support for the 'all' value to allowed_domains
Add tests to check previous behavior of allowed_domains and to test the case when using the 'all' keyword, refactor code a bit based in feedback
Refactor allow_domains feature, use better name for tests and change behavior to allow any domain when variable is set to empty list
Refactor allow_domains feature, use better name for tests and change behavior to allow any domain when variable is set to empty list