performance improvement to cookie handling when a single spider is crawling a large number of domains
Explicitly check if an object is a class is pydispatch.

* This is to fix a disagreement in between CPython and PyPy.
* https://gist.github.com/4220533
PEP8 on pydispatch/robustapply.py
doc: Response.replace() cannot take meta argument

>>> response.replace(meta={'foo':1})
Traceback (most recent call last):
  File "<input>", line 1, in <module>
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/text.py", line 45, in replace
    return Response.replace(self, *args, **kwargs)
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/__init__.py", line 77, in replace
    return cls(*args, **kwargs)
  File "/srv/scrapy/scrapy-fork/scrapy/scrapy/http/response/text.py", line 22, in __init__
    super(TextResponse, self).__init__(*args, **kwargs)
TypeError: __init__() got an unexpected keyword argument 'meta'
doc: remove trailing spaces
ported code to use queuelib
Merge pull request #77 from shane42/master

Cookie handling performance improvement
rfc2965 is dead

- It is not enabled by default in python cookielib
- Mozilla rejected implementing it https://bugzilla.mozilla.org/show_bug.cgi?id=208985
- Netscape cookies still rules
- It was superseded by RFC6265 which is the facto protocol formalized
cookies: increase candidate list with dot prefixed domains
fix broken doctests
remove scrapyd doc, as it was moved to its own repo
bind mockserver to 0.0.0.0, to listen on all 127.* range (useful for testing broad crawls)
Add garbage collector to downloader

This fixes a couple of issues:
- reactor callLater leaks when using download delay (test was
  re-enabled)
- downloader slot leaking on broad crawls (slots were created but never
  removed)
Merge pull request #297 from scrapy/downloader-gc

Add garbage collector to downloader
removed obsolete (and broken) downloader method: is_idle()
mention crawlera in best practices, as a way to deal with bans
Use twisted.web.client.Agent for download requests (use of HTTP/1.1)

Adds http11.HttpDownloadHandler in scrapy.core.downloader.handlers
Renamed downloader to Http11DownloadHandler and some refactoring

Only for HTTP, not HTTPS
Test on expected body length instead of request method (HEAD case)
Restore handling of HTTPS
http11 cleanup