scrapy/scrapy — pull request #714 by andrewbaxter (2014-05-07 10:21:36+00:00)
Title: Elaborated Request priority value in Scrapy docs

-priority is used to add the requests to a queuelib PriorityQueue

I tested with this code (n=2 mostly occurred first, followed by n=1 and then n=0):
from scrapy.spider import Spider
from random import shuffle
from scrapy.http import Request

class TestSpider(Spider):
name = "test"
start_urls = [
'http://www.linkedin.com/company/scrapinghub?n=8',
'http://www.linkedin.com/company/scrapinghub?n=9'
]

def parse(self, response):
reqs = [Request(
'http://www.linkedin.com/company/scrapinghub?n=0',
dont_filter=True,
priority = -50,
callback=self.parse2
) for i in range(10)] + [Request(
'http://www.linkedin.com/company/scrapinghub?n=1',
dont_filter=True,
priority = 0,
callback=self.parse2
) for i in range(10)] + [Request(
'http://www.linkedin.com/company/scrapinghub?n=2',
dont_filter=True,
priority = 50,
callback=self.parse2
) for i in range(10)]
shuffle(reqs)
return reqs

def parse2(self, response):
pass