Package: scrapingant-client
ScrapingAnt API client for Pythonscrapingant-clientis the official library to accessScrapingAnt APIfrom your Python
applications. It provides useful features like parameters encoding to improve the ScrapingAnt usage experience. Requires
python 3.6+.Quick StartAPI tokenAPI ReferenceExceptionsExamplesUseful linksQuick Startfromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')# Scrape the example.com siteresult=client.general_request('https://example.com')print(result.content)Installpipinstallscrapingant-clientIf you need async support:pipinstallscrapingant-client[async]API tokenIn order to get API token you'll need to register atScrapingAnt ServiceAPI ReferenceAll public classes, methods and their parameters can be inspected in this API reference.ScrapingAntClient(token)Main class of this library.ParamTypetokenstringScrapingAntClient.general_request and ScrapingAntClient.general_request_asynchttps://docs.scrapingant.com/request-response-format#available-parametersParamTypeDefaulturlstringmethodstringGETcookiesList[Cookie]NoneheadersList[Dict[str, str]]Nonejs_snippetstringNoneproxy_typeProxyTypedatacenterproxy_countrystrNonewait_for_selectorstrNonebrowserbooleanTruereturn_page_sourcebooleanFalsedatasame asrequests param 'data'Nonejsonsame asrequests param 'json'NoneIMPORTANT NOTE:js_snippetwill be encoded to Base64 automatically by the ScrapingAnt client library.CookieClass defining cookie. Currently it supports only name and valueParamTypenamestringvaluestringResponseClass defining response from API.ParamTypecontentstringcookiesList[Cookie]status_codeinttextstringExceptionsScrapingantClientExceptionis base Exception class, used for all errors.ExceptionReasonScrapingantInvalidTokenExceptionThe API token is wrong or you have exceeded the API calls request limitScrapingantInvalidInputExceptionInvalid value provided. Please, look into error message for more infoScrapingantInternalExceptionSomething went wrong with the server side code. Try again later or contact ScrapingAnt supportScrapingantSiteNotReachableExceptionThe requested URL is not reachable. Please, check it locallyScrapingantDetectedExceptionThe anti-bot detection system has detected the request. Please, retry or change the request settings.ScrapingantTimeoutExceptionGot timeout while communicating with Scrapingant servers. Check your network connection. Please try later or contact supportExamplesSending custom cookiesfromscrapingant_clientimportScrapingAntClientfromscrapingant_clientimportCookieclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')result=client.general_request('https://httpbin.org/cookies',cookies=[Cookie(name='cookieName1',value='cookieVal1'),Cookie(name='cookieName2',value='cookieVal2'),])print(result.content)# Response cookies is a list of Cookie objects# They can be used in next requestsresponse_cookies=result.cookiesExecuting custom JS snippetfromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')customJsSnippet="""var str = 'Hello, world!';var htmlElement = document.getElementsByTagName('html')[0];htmlElement.innerHTML = str;"""result=client.general_request('https://example.com',js_snippet=customJsSnippet,)print(result.content)Exception handling and retriesfromscrapingant_clientimportScrapingAntClient,ScrapingantClientException,ScrapingantInvalidInputExceptionclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')RETRIES_COUNT=3defparse_html(html:str):...# Implement your data extraction hereparsed_data=Noneforretry_numberinrange(RETRIES_COUNT):try:scrapingant_response=client.general_request('https://example.com',)exceptScrapingantInvalidInputExceptionase:print(f'Got invalid input exception:{{repr(e)}}')break# We are not retrying if request params are not validexceptScrapingantClientExceptionase:print(f'Got ScrapingAnt exception{repr(e)}')exceptExceptionase:print(f'Got unexpected exception{repr(e)}')# please report this kind of exceptions by creating a new issueelse:try:parsed_data=parse_html(scrapingant_response.content)break# Data is parsed successfully, so we dont need to retryexceptExceptionase:print(f'Got exception while parsing data{repr(e)}')ifparsed_dataisNone:print(f'Failed to retrieve and parse data after{RETRIES_COUNT}tries')# Can sleep and retry later, or stop the script execution, and research the reasonelse:print(f'Successfully parsed data:{parsed_data}')Sending custom headersfromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')result=client.general_request('https://httpbin.org/headers',headers={'test-header':'test-value'})print(result.content)# Http basic auth exampleresult=client.general_request('https://jigsaw.w3.org/HTTP/Basic/',headers={'Authorization':'Basic Z3Vlc3Q6Z3Vlc3Q='})print(result.content)Simple async exampleimportasynciofromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')asyncdefmain():# Scrape the example.com siteresult=awaitclient.general_request_async('https://example.com')print(result.content)asyncio.run(main())Sending POST requestfromscrapingant_clientimportScrapingAntClientclient=ScrapingAntClient(token='<YOUR-SCRAPINGANT-API-TOKEN>')# Sending POST request with json dataresult=client.general_request(url="https://httpbin.org/post",method="POST",json={"test":"test"},)print(result.content)# Sending POST request with bytes dataresult=client.general_request(url="https://httpbin.org/post",method="POST",data=b'test_bytes',)print(result.content)Useful linksScrapingant API doumentationScrapingant JS Client