Package: scrapy-promise
scrapy-promisePromise API for making Scrapy requests.Usage & Examplesfromscrapy_promiseimportfetchThePromisehere works like Promise in JavaScript. If you are new to Promise, a great starting point would be MDN'sPromise API referenceand the guide toUsing Promises.Creating and making requestsfetch()accepts all arguments thatscrapy.http.Requestaccepts, except forcallbackanderrback>>>fetch('https://example.org/login',method='POST',meta={'username':'admin'})fetch()returns aPromiseobject, which is an iterator/generator. You can return it directly instart_requests,
oryield fromit in an existing callback.Adding handlersIf you only callfetch()and yield from it, then all it does is storing the response once the request is finished:request=fetch('https://httpbin.org/ip')yield fromrequest# When the request is done>>>request.is_fulfilledTrue>>>request.get()<200https://httpbin.org/ip>fetch()returns aPromiseobject. Call its.then()method and provide a callable, andPromisewill call it once there is a response..then()returns anotherPromisethat you canyield from:defon_fulfill(response:TextResponse):# You can yield items from your handler# just like you would in a Scrapy callbackyieldItem(response)>>>yield fromfetch(...).then(on_fulfill)You can also attach an error handler with.catch(), which will receive either a TwistedFailureor an Exception:defon_reject(exc:Union[Failure,Exception]):ifisinstance(exc,Failure):exc=exc.value...>>>yield fromfetch(...).then(on_fulfill).catch(on_reject)# will catch both exceptions during the request# and exceptions raised in on_fulfillBranching and chainingBecause.then()and.catch()return another Promise, you can chain additional handlers.Subsequent handlers
will receive thereturnvalue of the previous handler.This is different from an ordinary Scrapy callback,where returning a value has no effect:yield from(fetch('https://httpbin.org/ip').then(parse_json)# returns dict.then(create_item)# will be passed the dict from the previous handler.catch(lambdaexc:logging.getLogger().error(exc)))Dynamic chaining: If you return anotherfetch()request in your handler, that request will be scheduled,
and the next handler will be called with theResponseof this new request. This lets you schedule multiple
requests in order.yield from(fetch('https://httpbin.org/ip')# A second Request is created from the response of the first one and is scheduled..then(lambdaresponse:fetch(json.loads(response.text)['origin'])).then(lambdaresponse:(yieldItem(response))).catch(lambdaexc:logging.getLogger().error(exc)))Note that only the request youreturnedwill be connected to subsequent handlers,Requests that are yielded
in the middle of the handler will be scheduled directly by Scrapy.You can also attach multiple handlers to one request, and they will be evaluated in the order they were
declared:resource=fetch(...)resource.then(save_token)resource.then(parse_html).catch(log_error)resource.then(next_page).catch(stop_spider)yield fromresource# Evaluating any Promise in a chain/branch causes# the entire Promise tree to be evaluated.Promise aggregation functionsPromiseprovides several aggregation functions that let you better control the how the requests are scheduled.fromnotcallbackimportPromise# dependencyPromise.all()will only fulfill when all requests are made successfully, and will reject as soon as one of
the requests failed. If all the requests succeed, the handler will receive a list of Responses:defparse_pages(responses:Tuple[TextResponse]):forrinresponses:...yield fromPromise.all(*[fetch(url)forurlinurls]).then(parse_pages)Promise.race()will fulfill as soon as one of the requests is fulfilled/rejected.defselect_fastest_cdn():yield from(Promise.race(*[fetch(url,method='HEAD')forurlincdn_list]).then(crawl_server))Promise.all_settled()always fulfills when all requests are finished, regardless of whether or not
they are successful. The handler will receive a list ofPromises whose value (the response) can be accessed
with the.get()method:defreport(promises:Tuple[Promise]):forpromiseinpromises:result=promise.get()ifisinstance(result,Response):log.info(f'Crawled{result.url}')else:log.warn(f'Encountered error{result}')yield fromPromise.all_settled(*[fetch(u)foruinurls]).then(report)Promise.any()fulfills with the first request that fulfills, and rejects if no request is successful:defdownload(response):...yield from(Promise.any(*[fetch(u)foruinurls]).then(download).catch(lambdaexc:log.warn('No valid URL!')))For more info on thePromise API, seenotcallbackSee alsoOther ways to schedule requests within a callback:twisted.internet.defer.inlineCallbacks(f)scrapy-inline-requests