Package: scrapinghub-autoextract
Python client libraries forScrapinghub AutoExtract API.
It allows to extract product, article, job posting, etc.
information from any website - whatever the API supports.Command-line utility, asyncio-based library and a simple synchronous wrapper
are provided by this package.License is BSD 3-clause.Installationpip install scrapinghub-autoextractscrapinghub-autoextract requires Python 3.6+ for CLI tool and for
the asyncio API; basic, synchronous API works with Python 3.5.UsageFirst, make sure you have an API key. To avoid passing it inapi_keyargument with every call, you can setSCRAPINGHUB_AUTOEXTRACT_KEYenvironment variable with the key.Command-line interfaceThe most basic way to use the client is from a command line.
First, create a file with urls, an URL per line (e.g.urls.txt).
Second, setSCRAPINGHUB_AUTOEXTRACT_KEYenv variable with your
AutoExtract API key (you can also pass API key as--api-keyscript
argument).Then run a script, to get the results:python -m autoextract urls.txt --page-type article --output res.jlNoteThe results can be stored in an order which is different from the input
order. If you need to match the output results to the input URLs, the
best way is to usemetafield (see below); it is passed through,
and returned as-is inrow["query"]["userQuery"]["meta"].If you need more flexibility, you can customize the requests by creating
a JsonLines file with queries: a JSON object per line. You can pass any
AutoExtract options there. Example - store it inqueries.jlfile:{"url": "http://example.com", "meta": "id0", "articleBodyRaw": false}
{"url": "http://example.com/foo", "meta": "id1", "articleBodyRaw": false}
{"url": "http://example.com/bar", "meta": "id2", "articleBodyRaw": false}SeeAPI docsfor a description of all supported parameters in these query
dicts. API docs mention batch requests and their limitation
(no more than 100 queries at time); these limits don’t apply to the queries.jl
file (i.e. it may have millions of rows), as the command-line script does
its own batching.Note that in the examplepageTypeargument is omitted;pageTypevalues are filled automatically from--page-typecommand line argument
value. You can also set a differentpageTypefor a row inqueries.jlfile; it has a priority over--page-typepassed in cmdline.To get results for thisqueries.jlfile, run:python -m autoextract --intype jl queries.jl --page-type article --output res.jlProcessing speedEach API key has a limit on RPS. To get your URLs processed faster you can
tune concurrency options: batch size and a number of connections.Best options depend on the RPS limit and on websites you’re extracting
data from. For example, if your API key has a limit of 3RPS, and average
response time you observe for your websites is 10s, then to get to these
3RPS you may set e.g. batch size = 2, number of connections = 15 - this
would allow to process 30 requests in parallel.To set these options in the CLI, use--n-connand--batch-sizearguments:python -m autoextract urls.txt --page-type articles --n-conn 15 --batch-size 2 --output res.jlIf too many requests are being processed in parallel, you’ll be getting
throttling errors. They are handled by CLI automatically, but they make
extraction less efficient; please tune the concurrency options to
not hit the throttling errors (HTTP 429) often.You may be also limited by the website speed. AutoExtract tries not to hit
any individual website too hard, but it could be better to limit this on
a client side as well. If you’re extracting data from a single website,
it could make sense to decrease the amount of parallel requests; it can ensure
higher success ratio overall.If you’re extracting data from multiple websites, it makes sense to spread the
load across time: if you have websites A, B and C, don’t send requests in
AAAABBBBCCCC order, send them in ABCABCABCABC order instead.To do so, you can change the order of the queries in your input file.
Alternatively, you can pass--shuffleoptions; it randomly shuffles
input queries before sending them to the API:python -m autoextract urls.txt –shuffle –page-type articles –output res.jlRunpython-mautoextract--helpto get description of all supported
options.ErrorsThe following errors could happen while making requests:Network errorsRequest-level errorsAuthentication failureMalformed requestToo many queries in requestRequest payload size is too largeQuery-level errorsDownloader errorsProxy errors…Some errors can be retried while others can’t.For example,
you can retry a query with a Proxy Timeout error
because this is a temporary error
and there are chances that this response will be different
within the next retries.On the other hand,
it makes no sense to retry queries that return a 404 Not Found error
because the response is not supposed to change if retried.RetriesBy default, we will automatically retry Network and Request-level errors.
You could also enable Query-level errors retries
by specifying the--max-query-error-retriesargument.Enable Query-level retries to increase the success rate
at the cost of more requests being performed
if you are interested in a higher success rate.python -m autoextract urls.txt --page-type articles --max-query-error-retries 3 --output res.jlFailing queries are retried
until the max number of retries or a timeout is reached.
If it’s still not possible to fetch all queries without errors,
the last available result is written to the output
including both queries with success and the ones with errors.Synchronous APISynchronous API provides an easy way to try AutoExtract.
For production usage asyncio API is strongly recommended. Currently the
synchronous API doesn’t handle throttling errors, and has other limitations;
it is most suited for quickly checking extraction results for a few URLs.To send a request, userequest_rawfunction; consult with theAPI docsto understand how to populate the query:from autoextract.sync import request_raw
query = [{'url': 'http://example.com.foo', 'pageType': 'article'}]
results = request_raw(query)Note that if there are several URLs in the query, results can be returned in
arbitrary order.There is also aautoextract.sync.request_batchhelper, which accepts URLs
and page type, and ensures results are in the same order as requested URLs:from autoextract.sync import request_batch
urls = ['http://example.com/foo', 'http://example.com/bar']
results = request_batch(urls, page_type='article')NoteCurrently request_batch is limited to 100 URLs at time only.asyncio APIBasic usage is similar to the sync API (request_raw),
but asyncio event loop is used:from autoextract.aio import request_raw

async def foo():
    query = [{'url': 'http://example.com.foo', 'pageType': 'article'}]
    results1 = await request_raw(query)
    # ...There is alsorequest_parallel_as_completedfunction, which allows
to process many URLs in parallel, using both batching and multiple
connections:import sys
from autoextract.aio import request_parallel_as_completed, create_session
from autoextract import ArticleRequest

async def extract_from(urls):
    requests = [ArticleRequest(url) for url in urls]
    async with create_session() as session:
        res_iter = request_parallel_as_completed(requests,
                                    n_conn=15, batch_size=2,
                                    session=session)
        for fut in res_iter:
            try:
                batch_result = await fut
                for res in batch_result:
                    # do something with a result, e.g.
                    print(json.dumps(res))
            except RequestError as e:
                print(e, file=sys.stderr)
                raiserequest_parallel_as_completedis modelled afterasyncio.as_completed(seehttps://docs.python.org/3/library/asyncio-task.html#asyncio.as_completed),
and actually uses it under the hood.Notefrom autoextract import ArticleRequestand its usage in the
example above. There are several Request helper classes,
which simplify building of the queries.request_parallel_as_completedandrequest_rawfunctions handle
throttling (http 429 errors) and network errors, retrying a request in
these cases.CLI interface implementation (autoextract/__main__.py) can serve
as an usage example.Request helpersTo query AutoExtract you need to create a dict with request parameters, e.g.:{'url': 'http://example.com.foo', 'pageType': 'article'}To simplify the library usage and avoid typos, scrapinghub-autoextract
provides helper classes for constructing these dicts:* autoextract.Request
* autoextract.ArticleRequest
* autoextract.ProductRequest
* autoextract.JobPostingRequestYou can pass instances of these classes instead of dicts everywhere when
requests dicts are accepted. So e.g. instead of writing this:query = [{"url": url, "pageType": "article"} for url in urls]You can write this:query = [Request(url, pageType="article") for url in urls]or this:query = [ArticleRequest(url) for url in urls]There is one difference:articleBodyRawparameter is set toFalseby default when Request or its variants are used, while it isTrueby default in the API.You can override API params passing a dictionary with extra data using theextraargument. Note that it will overwrite any previous configuration
made using standard attributes likearticleBodyRawandfullHtml.Extra parameters example:request = ArticleRequest(
    url=url,
    fullHtml=True,
    extra={
        "customField": "custom value",
        "fullHtml": False
    }
)This will generate a query that looks like this:{
    "url": url,
    "pageType": "article",
    "fullHtml": False,  # our extra parameter overrides the previous value
    "customField": "custom value"  # not a default param but defined even then
}ContributingSource code:https://github.com/scrapinghub/scrapinghub-autoextractIssue tracker:https://github.com/scrapinghub/scrapinghub-autoextract/issuesUsetoxto run tests with different Python versions:toxThe command above also runs type checks; we use mypy.Changes0.6.1 (2021-01-27)fixedmax_retriesbehaviour. Total attempts must be max_retries + 10.6.0 (2020-12-29)CLI changes: error display in the progress bar is changed;
summary is printed after the executionsmore errors are retried when retrying is enabled, which allows for a higher
success ratefixed tcp connection poolingautoextract.aio.request_rawfunction allows to pass custom headers
to the API (not to remote websites)autoextract.aio.request_rawnow allows to customize the retry
behavior, viaretryingargumenttenacity.RetryErroris no longer raised by the library; concrete errors
are raised insteadPython 3.9 supportCI is moved from Travis to Github Actions0.5.2 (2020-11-27)QueryErroris renamed to_QueryError, as this is not an error
users of the library ever see.Retrials were broken by having userAgent in the userQuery API output;
temporary workaround is added to make retrials work again.0.5.1 (2020-08-21)fix a problem that was preventing calls torequest_rawwhenendpointargument wasNone0.5.0 (2020-08-21)add--api-endpointoption to the command line utilityimproves documentation adding details aboutRequest’s extra parameters0.4.0 (2020-08-17)autoextract.Requesthelper class now allows to set arbitrary
parameters for AutoExtract requests - they can be passed inextraargument.0.3.0 (2020-07-24)In this release retry-related features are added or improved.
It is now possible to fix some of the temporary errors
by enabling query-level retries, and the default retry behavior is improved.backwards-incompatible: autoextract.aio.ApiError is renamed
to autoextract.aio.RequestErrormax_query_error_retriesargument is added toautoextract.aio.request_rawandautoextract.aio.request_parallel_as_completedfunctions; it allows to
enable retries of temporary query-level errors returned by the API.CLI: added--max-query-error-retriesoption to retry temporary
query-level errors.HTTP 500 errors from server are retried now;documentation and test improvements.0.2.0 (2020-04-15)asyncio API is rewritten, to simplify use in cases where passing meta
is required.autoextract.aio.request_parallel_as_completedis added,autoextract.aio.request_parallelandautoextract.aio.request_batchare removed.CLI: it now shows various stats: mean response and connect time,
% of throttling errors, % of network and other errorsCLI: new--intypejloption allows to process a .jl file
with arbitrary AutoExtract API queriesCLI: new--shuffleoption allows to shuffle input data, to spread it
more evenly across websites.CLI: it no longer exits on unrecoverable errors, to aid long-running
processing tasks.retry logic is adjusted to handle network errors better.autoextract.aio.request_rawandautoextract.aio.request_parallel_as_completedfunctions provide an
interface to return statistics about requests made, including retries.autoextract.Request, autoextract.ArticleRequest, autoextract.ProductRequest,
autoextract.JobPostingRequest helper classesDocumentation improvements.0.1.1 (2020-03-12)allow up to 100 elements in a batch, not up to 99custom User-Agent header is addedPython 3.8 support is declared & tested0.1 (2019-10-09)Initial release.