Package: scrapy-autoextract
This library integrates Zyte’s AI Enabled Automatic Data Extraction
into a Scrapy spider by two different means:with a downloader middleware that injects the AutoExtract responses intoresponse.meta['autoextract']for consumption by the spider.with ascrapy-poetprovider that injects the responses as callback parameters.Installationpip install scrapy-autoextractscrapy-autoextract requires Python 3.6+ for the download middleware and Python 3.7+ for the scrapy-poet providerUsageThere are two different ways to consume the AutoExtract API with this library:using our Scrapy middlewareusing our Page Object providerThe middlewareThe middleware is opt-in and can be explicitly enabled per request,
with the{'autoextract': {'enabled': True}}request meta.
All the options below can be set either in the project settings file,
or just for specific spiders, in thecustom_settingsdict.Within the spider, consuming the AutoExtract result is as easy as:def parse(self, response):
    yield response.meta['autoextract']ConfigurationAdd the AutoExtract downloader middleware in the settings file:DOWNLOADER_MIDDLEWARES = {
    'scrapy_autoextract.AutoExtractMiddleware': 543,
}Note that this should be the last downloader middleware to be executed.The providersAnother way of consuming AutoExtract API is using the Page Objects pattern
proposed by theweb-poetlibrary and implemented byscrapy-poet.Items returned by Page Objects are defined in theautoextract-poetlibrary.Within the spider, consuming the AutoExtract result is as easy as:import scrapy
from autoextract_poet.pages import AutoExtractArticlePage

class SampleSpider(scrapy.Spider):
    name = "sample"

    def parse(self, response, article_page: AutoExtractArticlePage):
        # We're making two requests here:
        # - one through Scrapy to build the response argument
        # - the other through the providers to build the article_page argument
        yield article_page.to_item()Note that on the example above, we’re going to perform two requests:one goes through Scrapy (it might use Smart Proxy, Splash or no proxy at all, depending on your configuration)another goes through AutoExtract API usingzyte-autoextractIf you don’t need the additional request going through Scrapy,
you can annotate the response argument of your callback withDummyResponse.
This will ignore the Scrapy request and only the AutoExtract API will be fetched.For example:import scrapy
from autoextract_poet.pages import AutoExtractArticlePage
from scrapy_poet import DummyResponse

class SampleSpider(scrapy.Spider):
    name = "sample"

    def parse(self, response: DummyResponse, article_page: AutoExtractArticlePage):
        # We're making a single request here to build the article argument
        yield article_page.to_item()The examples above extract an article from the page, but you may want to
extract a different type of item, like a product or a job posting. It is
as easy as using the correct type annotation in the callback. This
is how the callback looks like if we need to extract a real state
from the page:def parse(self,
          response: DummyResponse,
          real_estate_page: AutoExtractRealEstatePage):
    yield real_estate_page.to_item()You can even useAutoExtractWebPageif what you need is the raw browser HTML to
extract some additional data. Visit the full list ofsupported page typesto get a better idea of the supported pages.ConfigurationFirst, you need to configure scrapy-poet as described onscrapy-poet’s documentationand then enable AutoExtract providers by putting the following code to Scrapy’ssettings.pyfile:# Install AutoExtract provider
SCRAPY_POET_PROVIDERS = {"scrapy_autoextract.AutoExtractProvider": 500}

# Enable scrapy-poet's provider injection middleware
DOWNLOADER_MIDDLEWARES = {
    'scrapy_poet.InjectionMiddleware': 543,
}

# Configure Twisted's reactor for asyncio support on Scrapy
TWISTED_REACTOR = 'twisted.internet.asyncioreactor.AsyncioSelectorReactor'Currently, our providers are implemented using asyncio.
Scrapy has introduced asyncio support since version 2.0
but as of Scrapy 2.3 you need to manually enable it by configuring Twisted’s default reactor.
CheckScrapy’s asyncio documentationfor more information.Checklist:scrapy-poet is installed and downloader/injector middleware is configuredautoextract-poet is installed (page inputs are imported from this lib)providers are configured on settings.pyScrapy’s asyncio support is enabled on settings.pyNow you should be ready to use our AutoExtract providers.ExceptionsWhile trying to fetch AutoExtract API, providers might raise some exceptions.
Those exceptions might come from scrapy-autoextract providers themselves,zyte-autoextract, or by other means (e.g.ConnectionError).
For example:autoextract.aio.errors.RequestError: raised when aRequest-level erroris returnedscrapy_autoextract.errors.QueryError: raised when aQuery-level erroris returnedCheckzyte-autoextract’s async errorsfor other exception definitions.You can capture those exceptions using an error callback (errback):import scrapy
from autoextract.aio.errors import RequestError
from autoextract_poet.pages import AutoExtractArticlePage
from scrapy_autoextract.errors import QueryError
from scrapy_poet import DummyResponse
from twisted.python.failure import Failure

class SampleSpider(scrapy.Spider):
    name = "sample"
    urls = [...]

    def start_requests(self):
        for url in self.urls:
            yield scrapy.Request(url, callback=self.parse_article,
                                 errback=self.errback_article)

    def parse_article(self, response: DummyResponse,
                      article_page: AutoExtractArticlePage):
        yield article_page.to_item()

    def errback_article(self, failure: Failure):
        if failure.check(RequestError):
            self.logger.error(f"RequestError on {failure.request.url}")

        if failure.check(QueryError):
            self.logger.error(f"QueryError: {failure.value.message}")SeeScrapy documentationfor more details on how to capture exceptions using request’s errback.SettingsMiddleware settingsAUTOEXTRACT_USER[mandatory] is your AutoExtract API keyAUTOEXTRACT_URL[optional] the AutoExtract service url. Defaults to autoextract.scrapinghub.com.AUTOEXTRACT_TIMEOUT[optional] sets the response timeout from AutoExtract. Defaults to 660 seconds.
Can also be defined by setting the “download_timeout” in the request.meta.AUTOEXTRACT_PAGE_TYPE[mandatory] defines the kind of document to be extracted.
Current available options are“product”and“article”.
Can also be defined onspider.page_type, or{'autoextract': {'pageType':'...'}}request meta.
This is required for the AutoExtract classifier to know what kind of page needs to be extracted.extra[optional] allows sending extra payload data to your AutoExtract request.
Must be specified as{'autoextract': {'extra':{}}}request meta and must be a dict.AUTOEXTRACT_SLOT_POLICY[optional] Download concurrency options. Defaults toSlotPolicy.PER_DOMAIN- If set toSlotPolicy.PER_DOMAIN, then consider settingSCHEDULER_PRIORITY_QUEUE = 'scrapy.pqueues.DownloaderAwarePriorityQueue'to make better usage of AutoExtract concurrency and avoid delays.Provider settingsAUTOEXTRACT_USER[optional] is your AutoExtract API key. If not set, it is
taken from ZYTE_AUTOEXTRACT_KEY environment variable.AUTOEXTRACT_URL[optional] the AutoExtract service url.
Defaults to the official AutoExtract endpoint.AUTOEXTRACT_MAX_QUERY_ERROR_RETRIES[optional] Max number of retries for
Query-level errors. Defaults to0.AUTOEXTRACT_CONCURRENT_REQUESTS_PER_DOMAIN[optional] Max number
of concurrent requests per domain. If not set, the provider will search
for theCONCURRENT_REQUESTS_PER_DOMAIN(defaults to8) setting instead.AUTOEXTRACT_CACHE_FILENAME[optional] Filename of a .sqlite file that will
be placed in the.scrapyfolder. File will be created if it doesn’t exist.
Cache is useful for development; AutoExtract requests bypass standard Scrapy
cache when providers are used.LimitationsWhen using the AutoExtract middleware, there are some limitations.The incoming spider request is rendered by AutoExtract, not just downloaded by Scrapy,
which can change the result - the IP is different, headers are different, etc.Only GET requests are supportedCustom headers and cookies are not supported (i.e. Scrapy features to set them don’t work)Proxies are not supported (they would work incorrectly,
sitting between Scrapy and AutoExtract, instead of AutoExtract and website)AutoThrottle extension can work incorrectly for AutoExtract requests,
because AutoExtract timing can be much larger than time required to download a page,
so it’s best to useAUTHTHROTTLE_ENABLED=Falsein the settings.Redirects are handled by AutoExtract, not by Scrapy,
so these kinds of middlewares might have no effect429 errors could be handled as standard retries when using Scrapy middleware,
but they’re handled properly and automatically with scrapy-poet integration,
as it relies onzyte-autoextract.
You may lose some responses with the middleware approach.Overall, retries have a better behavior with scrapy-poet integration
and it includes support for automatic Query-level errors retries with
no need to changeRETRY_HTTP_CODES.AutoExtract-specific cache (AUTOEXTRACT_CACHE_FILENAME) is not supportedWhen using the AutoExtract providers, be aware that:With scrapy-poet integration, retry requests don’t go through Scrapy