Package: zyte-api-convertor
zyte-api-convertorA Python module to convert Zyte API Json payload toScrapy ZyteAPIproject.
It uses Scrapy and scrapy-zyte-api plugin to generate the project, also it uses black to format the code.RequirementsPython 3.6+
Scrapy
scrapy-zyte-api
blackDocumentationZyte API DocumentationTest the Zyte API payload using postman or curl. Once it gives the desired response, use the same payload with this module to convert it to a Scrapy ZyteAPI project.Installationpip install zyte-api-convertorUsageUsage:zyte-api-convertor<payload>--project-name<project_name>--spider-name<spider_name>Example:zyte-api-convertor'{"url": "https://httpbin.org/ip", "browserHtml": true, "screenshot": true}'--project-namesample_project--spider-namesample_spiderUsage:zyte-api-convertor<payload>--project-name<project_name>Example:zyte-api-convertor'{"url": "https://httpbin.org/ip", "browserHtml": true, "screenshot": true}'--project-namesample_projectUsage:zyte-api-convertor<payload>--spider-name<spider_name>Example:zyte-api-convertor'{"url": "https://httpbin.org/ip", "browserHtml": true, "screenshot": true}'--spider-namesample_spiderUsage:zyte-api-convertor<payload>Example:zyte-api-convertor'{"url": "https://httpbin.org/ip", "browserHtml": true, "screenshot": true}'Examplezyte-api-convertor expects a valid json payload at the least. But it does have other options as well. You can use the--project-nameand--spider-nameoptions to set the project and spider name. If you don't use these options, it will use the default project and spider name.zyte-api-convertor'{"url": "https://httpbin.org/ip", "browserHtml": true, "screenshot": true}'--project-namesample_project--spider-namesample_spiderOutput:mukthy@Mukthys-MacBook-Pro%zyte-api-convertor'{"url": "https://httpbin.org/ip", "browserHtml": true, "screenshot": true}'--project-namesample_project--spider-namesample_spider
CodeGenerated!
Writingtofile...
WritingDone!
reformattedsample_project/sample_project/spiders/sample_project.py

Alldone!✨🍰✨1filereformatted.
FormattingDone!Project Created Successfully.mukthy@Mukthys-MacBook-Pro%sample_project%tree
.
├──sample_project
│├──__init__.py
│├──items.py
│├──middlewares.py
│├──pipelines.py
│├──settings.py
│└──spiders
│├──__init__.py
│└──sample_project.py
└──scrapy.cfg3directories,8filesSample Spider Code:importscrapyclassSampleQuotesSpider(scrapy.Spider):name="sample_spider"custom_settings={"DOWNLOAD_HANDLERS":{"http":"scrapy_zyte_api.ScrapyZyteAPIDownloadHandler","https":"scrapy_zyte_api.ScrapyZyteAPIDownloadHandler",},"DOWNLOADER_MIDDLEWARES":{"scrapy_zyte_api.ScrapyZyteAPIDownloaderMiddleware":1000},"REQUEST_FINGERPRINTER_CLASS":"scrapy_zyte_api.ScrapyZyteAPIRequestFingerprinter","TWISTED_REACTOR":"twisted.internet.asyncioreactor.AsyncioSelectorReactor","ZYTE_API_KEY":"YOUR_API_KEY",}defstart_requests(self):yieldscrapy.Request(url="https://httpbin.org/ip",meta={"zyte_api":{"javascript":False,"screenshot":True,"browserHtml":True,"actions":[],"requestHeaders":{},"geolocation":"US","experimental":{"responseCookies":False},}},)defparse(self,response):print(response.text)Please note that theZYTE_API_KEYis not set in thecustom_settingsof the spider. You need to set it before running it.