CrawSpider: support process_links as generator

Adding tests for CrawlSpider's process_links
DupeFilter: add setting for verbose logging + stats counter for filtered requests
Remove _log_level attribute as per comments
Fixed #441.
Merge pull request #555 from redapple/crawlspider-processlinks

CrawlSpider: support process_links as generator
Merge pull request #553 from redapple/dupefilter-verbose

DupeFilter: add settings for verbose logging and filtered requests stats
fixes dynamic itemclass example usage of type()
Merge pull request #603 from barraponto/dynamic-itemclass-docfix

fixes dynamic itemclass example usage of type()
rename attribute to match conventions used for XXX_DEBUG settings (in autothrottle and cookies mw)
remove no longer used setting: MAIL_DEBUG
Merge pull request #602 from Blender3D/email

Added a mimetype parameter to `MailSender.send`
Update httperror.py

Log ignored HTTP pages when the response code is not allowed or not configured to be handled.
Update scrapy command line doc with additional scrapy parse options
remove unused setting: DOWNLOADER_DEBUG
Added missing word in practices.rst
Clarify MapCompose documentation
Fix some typos, whitespace and small errors in docs
Merge pull request #618 from allait/master

Fix some small typos and errors in docs
Generalize the file pipeline log messages so they are not specific to downloading images.
Merge pull request #622 from ajw0100/generalize-file-pipeline-log-messages

Generalize the file pipeline log messages