avoid exception with responses which has headers with no Content-Type
Merge pull request #61 from kalessin/master

avoid exception with responses which has headers with no Content-Type
Fix exception when Content-Type is empty and response type is not Html
Merge pull request #62 from andresp99999/fix-content-type

Fix exception when Content-Type is empty and response type is not Html
Fix small typo
Merge pull request #63 from amferraz/patch-1

Fix small typo
pass explicit encoding in url processors. Added testcase which motivated the fix.
Merge pull request #64 from kalessin/encoding

pass explicit encoding in url processors. Added testcase which motivated the fix.
Add support to define allowed_domains inside the spider definition
Merge pull request #59 from andresp99999/generic-form-implementation

Add allowed_domains variable to spider definition
add script for easily run a slybot spider, and some README improvements
and adds