Issue 1:
Hi,

I'm trying to perform a broad crawl of the web with Scrapy in breadth-first order.

The issue I'm running into is that after a few seconds of the crawl running, it seems to get stuck on just one or two domains instead of continuing down the list of seed URLs. I would expect it to either continue through the entire list of separate domains, and/or crawl multiple domains concurrently.

Output of scrapy version -v:

Scrapy : 1.3.2
lxml : 3.4.0.0
libxml2 : 2.9.1
cssselect : 1.0.1
parsel : 1.1.0
w3lib : 1.17.0
Twisted : 17.1.0
Python : 2.7.9 (default, Jun 29 2016, 13:08:31) - [GCC 4.9.2]
pyOpenSSL : 16.2.0 (OpenSSL 1.0.1t 3 May 2016)
Platform : Linux-3.16.0-4-amd64-x86_64-with-debian-8.7

Here is the main code for my spider:

# MySpider defines our Scrapy spider.
class MySpider(CrawlSpider):
name = "MyBot"
rules = (
Rule(LinkExtractor(deny_domains=deny_domains), callback='parse_item', follow=True),
)

def start_requests(self):
# Open the seed.csv file.
seed_path = os.path.dirname(os.path.realpath(__file__)) + '/seed.csv'
f = open(seed_path)
csv_rdr = csv.reader(f)

for row in csv_rdr:
url = 'http://' + row[0].strip()
yield Request(url=url, callback=self.parse)

def parse_start_url(self, response):
return self.parse_item(response)

def parse_item(self, response):
print '!!!!!!!!!!!!! Parsing: %s !!!!!!!!!!!!!' % response.url

# Check the Content-Type.
if is_content_type_ok(response.headers.getlist('Content-Type')):
# Yield data here
yield {}

Here is my settings.py file:

BOT_NAME = 'MyBot'

SPIDER_MODULES = ['crawler.spiders']
NEWSPIDER_MODULE = 'crawler.spiders'

# Crawl responsibly by identifying yourself (and your website) on the user-agent
USER_AGENT = 'BotTest (+https://test.com)'

# Obey robots.txt rules
ROBOTSTXT_OBEY = True

# Broad crawl settings.
CONCURRENT_REQUESTS = 128
REACTOR_THREADPOOL_MAXSIZE = 30
CONCURRENT_REQUESTS_PER_DOMAIN = 1
CONCURRENT_REQUESTS_PER_IP = 1
COOKIES_ENABLED = False
RETRY_ENABLED = False
DOWNLOAD_TIMEOUT = 3
LOG_LEVEL = 'INFO'
AJAXCRAWL_ENABLED = True
HTTPCACHE_ENABLED = False
LOGSTATS_INTERVAL = 10

# Disable auto throttling.
AUTOTHROTTLE_ENABLED = False

# Set delay.
RANDOMIZE_DOWNLOAD_DELAY = False
DOWNLOAD_DELAY = 2

# Set max download size to 2MB
DOWNLOAD_MAXSIZE = 2 * 1024 * 1024
DOWNLOAD_WARNSIZE = 0

# Set scheduler to BFO.
DEPTH_PRIORITY = 1
SCHEDULER_DISK_QUEUE = 'scrapy.squeues.PickleFifoDiskQueue'
SCHEDULER_MEMORY_QUEUE = 'scrapy.squeues.FifoMemoryQueue'

And here is example output from a crawl:

$:~/crawler$ scrapy crawl MyBot -s JOBDIR=crawls/crawl-1 -o output.csv -t csv
2017-03-04 22:56:19 [scrapy.utils.log] INFO: Scrapy 1.3.2 started (bot: MyBot)
2017-03-04 22:56:19 [scrapy.utils.log] INFO: Overridden settings: {'FEED_URI': 'output.csv', 'AJAXCRAWL_ENABLED': True, 'COOKIES_ENABLED': False, 'FEED_FORMAT': 'csv', 'DOWNLOAD_DELAY': 2, 'LOG_LEVEL': 'INFO', 'RETRY_ENABLED': False, 'CONCURRENT_REQUESTS_PER_IP': 1, 'DOWNLOAD_TIMEOUT': 3, 'LOGSTATS_INTERVAL': 10, 'DEPTH_PRIORITY': 1, 'SCHEDULER_MEMORY_QUEUE': 'scrapy.squeues.FifoMemoryQueue', 'SCHEDULER_DISK_QUEUE': 'scrapy.squeues.PickleFifoDiskQueue', 'CONCURRENT_REQUESTS': 128, 'RANDOMIZE_DOWNLOAD_DELAY': False, 'DOWNLOAD_WARNSIZE': 0, 'SPIDER_MODULES': ['crawler.spiders'], 'BOT_NAME': 'Bot', 'NEWSPIDER_MODULE': 'crawler.spiders', 'ROBOTSTXT_OBEY': True, 'CONCURRENT_REQUESTS_PER_DOMAIN': 1, 'REACTOR_THREADPOOL_MAXSIZE': 30, 'DOWNLOAD_MAXSIZE': 2097152}
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.spiderstate.SpiderState']
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.robotstxt.RobotsTxtMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.ajaxcrawl.AjaxCrawlMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2017-03-04 22:56:19 [scrapy.middleware] INFO: Enabled item pipelines:
[]
2017-03-04 22:56:19 [scrapy.core.engine] INFO: Spider opened
2017-03-04 22:56:19 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
!!!!!!!!!!!!! Parsing: http://imgur.com !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://api.imgur.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.sina.com.cn/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://twitter.com/imgur !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.msn.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.ask.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.apple.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.bbc.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.adobe.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://wordpress.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.tumblr.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.reddit.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.yahoo.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://www.imdb.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.aliexpress.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.youtube.com/ !!!!!!!!!!!!!

... seconds later ...

!!!!!!!!!!!!! Parsing: https://www.paypal.com/us/home !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://stackoverflow.blog/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://www.amazon.co.jp/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/privacy !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/tags !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://login.live.com/login.srf?wa=wsignin1.0&rpsnv=13&ct=1488668190&rver=6.4.6456.0&wp=MBI_SSL_SHARED&wreply=https:%2F%2Fmail.live.com%2Fdefault.aspx%3Frru%3Dinbox&lc=1033&id=64855&mkt=en-US&cbcxt=mai !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/upload !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/vidgif !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackexchange.com/sites !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/tour !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/memegen !!!!!!!!!!!!!
2017-03-04 22:56:39 [scrapy.extensions.logstats] INFO: Crawled 165 pages (at 90 pages/min), scraped 19 items (at 36 items/min)
!!!!!!!!!!!!! Parsing: http://meta.stackoverflow.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://imgur.com/signin?invokedBy=regularSignIn !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: https://imgur.com/register?invokedBy=regularSignIn !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/help !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/company/about !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/hot/viral !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackexchange.com/ !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/new/viral !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/?tab=interesting !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Funny !!!!!!!!!!!!!
2017-03-04 22:56:49 [scrapy.extensions.logstats] INFO: Crawled 175 pages (at 60 pages/min), scraped 25 items (at 36 items/min)
!!!!!!!!!!!!! Parsing: http://imgur.com/t/The_More_You_Know !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/?tab=featured !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Science_and_Tech !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/?tab=hot !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Gaming !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/?tab=week !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/?tab=month !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Eat_What_You_Want !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/questions/42602342/java-unlimited-cryptography-extension-doesnt-work !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Aww !!!!!!!!!!!!!
2017-03-04 22:56:59 [scrapy.extensions.logstats] INFO: Crawled 185 pages (at 60 pages/min), scraped 42 items (at 102 items/min)
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Inspiring !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/questions/tagged/authentication !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/questions/tagged/hdfs !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Awesome !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/questions/tagged/kerberos !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Creativity !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/questions/tagged/jce !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/The_Great_Outdoors !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://imgur.com/t/Storytime !!!!!!!!!!!!!
!!!!!!!!!!!!! Parsing: http://stackoverflow.com/users/3593261/user3593261 !!!!!!!!!!!!!
2017-03-04 22:57:09 [scrapy.extensions.logstats] INFO: Crawled 195 pages (at 60 pages/min), scraped 48 items (at 36 items/min)

Thanks, let me know if you need any other information!

Issue 2:
We need to add a deployment section to the documentation covering scrapyd-deploy.