Package: pa-scraper
Prompt API - Scraper API - Python Packagepa-scraperis a python wrapper forscraper apiwith few
more extra cream and sugar.RequirementsYou need to signup forPrompt APIYou need to subscribescraper api, test drive isfree!!!You need to setPROMPTAPI_TOKENenvironment variable after subscription.then;$pipinstallpa-scraperExample UsageExamples can be foundhere.# examples/fetch.pyfromscraperimportScraperurl='https://pypi.org/classifiers/'scraper=Scraper(url)response=scraper.get()ifresponse.get('error',None):# response['error']  returns error message# response['status'] returns http status code# Example: {'error': 'Not Found', 'status': 404}print(response)# noqa: T001else:data=response['result']['data']headers=response['result']['headers']url=response['result']['url']status=response['status']# print(data) # print fetched html, will be long :)print(headers)# noqa: T001# {'Content-Length': '321322', 'Content-Type': 'text/html; charset=UTF-8', ... }print(status)# noqa: T001# 200save_result=scraper.save('/tmp/my-data.html')# noqa: S108ifsave_result.get('error',None):# save error occured...# add you code here...passprint(save_result)# noqa: T001# {'file': '/tmp/my-data.html', 'size': 321322}You can add url parameters for extra operations. Valid parameters are:auth_password: for HTTP Realm auth passwordauth_username: for HTTP Realm auth usernamecookie: URL Encoded cookie header.country: 2 character country code. If you wish to scrape from an IP address of a specific country.referer: HTTP referer headerselector: CSS style selector path such asa.btn div li. Ifselectoris enabled, returning result will be collection of data and saved file
will be in.jsonformat.Here is an example with using url parameters andselector:# examples/fetch_with_params.pyfromscraperimportScraperurl='https://pypi.org/classifiers/'scraper=Scraper(url)fetch_params=dict(country='EE',selector='ul li button[data-clipboard-text]')response=scraper.get(params=fetch_params)ifresponse.get('error',None):# response['error']  returns error message# response['status'] returns http status code# Example: {'error': 'Not Found', 'status': 404}print(response)# noqa: T001else:data=response['result']['data']headers=response['result']['headers']url=response['result']['url']status=response['status']# print(data)  # noqa: T001# ['<button class="button button--small margin-top margin-bottom copy-tooltip copy-tooltip-w" ...\n', ]print(len(data))# noqa: T001# 734# we have an array...print(headers)# noqa: T001# {'Content-Length': '321322', 'Content-Type': 'text/html; charset=UTF-8', ... }print(status)# noqa: T001# 200save_result=scraper.save('/tmp/my-data.json')# noqa: S108ifsave_result.get('error',None):# save error occured...# add you code here...passprint(save_result)# noqa: T001# {'file': '/tmp/my-data.json', 'size': 174449}Defaulttimeoutvalue is set to10seconds. You can change this while
initializing the instance:scraper=Scraper(url,timeout=50)# 50 seconds timeout...You can also add custom headers prefixed withX-. Example below shows
how to add extra request headers and set default timeout:# pylint: disable=C0103fromscraperimportScraperif__name__=='__main__':url='https://pypi.org/classifiers/'scraper=Scraper(url)fetch_params=dict(country='EE',selector='ul li button[data-clipboard-text]')custom_headers={'X-Referer':'https://www.google.com','X-User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0',}timeout=30response=scraper.get(params=fetch_params,headers=custom_headers,timeout=timeout)ifresponse.get('error',None):# response['error']  returns error message# response['status'] returns http status code# Example: {'error': 'Not Found', 'status': 404}print(response)# noqa: T001else:data=response['result']['data']headers=response['result']['headers']url=response['result']['url']status=response['status']# print(data)  # noqa: T001# ['<button class="button button--small margin-top margin-bottom copy-tooltip copy-tooltip-w" ...\n', ]print(len(data))# noqa: T001# 734print(headers)# noqa: T001# {'Content-Length': '321322', 'Content-Type': 'text/html; charset=UTF-8', ... }print(status)# noqa: T001# 200save_result=scraper.save('/tmp/my-data.json')# noqa: S108ifsave_result.get('error',None):# save error occured...# add you code here...passprint(save_result)# noqa: T001# {'file': '/tmp/my-data.json', 'size': 174449}LicenseThis project is licensed under MITContributer(s)Prompt API- Creator, maintainerContributeAll PR’s are welcome!fork(https://github.com/promptapi/scraper-py/fork)Create yourbranch(git checkout -b my-feature)commityours (git commit -am 'Add awesome features...')pushyourbranch(git push origin my-feature)Than create a newPull Request!This project is intended to be a safe,
welcoming space for collaboration, and contributors are expected to adhere to
thecode of conduct.