Package: scrapy-ipfs-filecoin
Welcome to Scrapy-IPFS-FilecoinScrapy is a popular open-source and collaborative python framework for extracting the data you need from websites. scrapy-ipfs-filecoin provides scrapy pipelines and feed exports to store items into IPFS and Filecoin using services likeWeb3.Storage,LightHouse.Storage,Estuary,Pinata,Moralis,Filebaseor any S3 compatible services.🏠HomepageInstallnpminstall-ghttps://github.com/pawanpaudel93/ipfs-only-hash.gitpipinstallscrapy-ipfs-filecoinExamplescrapy-ipfs-filecoin-exampleUsageInstall ipfs-only-hash and scrapy-ipfs-filecoin.npminstall-ghttps://github.com/pawanpaudel93/ipfs-only-hash.gitpipinstallscrapy-ipfs-filecoinAdd 'scrapy-ipfs-filecoin.pipelines.ImagesPipeline' and/or 'scrapy-ipfs-filecoin.pipelines.FilesPipeline' to ITEM_PIPELINES setting in your Scrapy project if you need to store images or other files to IPFS and Filecoin.
For Images Pipeline, use:ITEM_PIPELINES={'scrapy_ipfs_filecoin.pipelines.ImagesPipeline':1}For Files Pipeline, use:ITEM_PIPELINES={'scrapy_ipfs_filecoin.pipelines.FilesPipeline':1}The advantage of using the ImagesPipeline for image files is that you can configure some extra functions like generating thumbnails and filtering the images based on their size.Or You can also use both the Files and Images Pipeline at the same time.ITEM_PIPELINES={'scrapy_ipfs_filecoin.pipelines.ImagesPipeline':1,'scrapy-ipfs-filecoin.pipelines.FilesPipeline':1}If you are using the ImagesPipeline make sure to install the pillow package. The Images Pipeline requires Pillow 7.1.0 or greater. It is used for thumbnailing and normalizing images to JPEG/RGB format.pipinstallpillowThen, configure the target storage setting to a valid value that will be used for storing the downloaded images. Otherwise the pipeline will remain disabled, even if you include it in the ITEM_PIPELINES setting.Add store path of files or images for Web3Storage, LightHouse, Moralis, Pinata or Estuary as required.# for ImagesPipelineIMAGES_STORE='w3s://images'# For Web3StorageIMAGES_STORE='es://images'# For EstuaryIMAGES_STORE='lh://images'# For LightHouseIMAGES_STORE='pn://images'# For PinataIMAGES_STORE='ms://images'# For MoralisIMAGES_STORE="s3://bucket-name/images/"# For Filebase or other s3 compatible services# For FilesPipelineFILES_STORE='w3s://files'# For Web3StorageFILES_STORE='es://files'# For EstuaryFILES_STORE='lh://files'# For LightHouseFILES_STORE='es://files'# For PinataFILES_STORE='pn://files'# For MoralisFILES_STORE="s3://bucket-name/files/"# For Filebase or other s3 compatible servicesFor more info regarding ImagesPipeline and FilesPipline.See hereFor Feed storage to store the output of scraping as json, csv, json, jsonlines, jsonl, jl, csv, xml, marshal, pickle etc set FEED_STORAGES as following for the desired output format:fromscrapy_ipfs_filecoin.feedexportimportget_feed_storagesFEED_STORAGES=get_feed_storages()Then set API Key for one of the storage i.e Web3Storage, LightHouse, Moralis, Pinata or Estuary. And, set FEEDS as following to finally store the scraped data.For Web3Storage:W3S_API_KEY="<W3S_API_KEY>"FEEDS={'w3s://house.json':{"format":"json"},}For LightHouse:LH_API_KEY="<LH_API_KEY>"FEEDS={'lh://house.json':{"format":"json"},}For Estuary:ES_API_KEY="<ES_API_KEY>"FEEDS={'es://house.json':{"format":"json"},}For Pinata:PN_JWT_TOKEN="<PN_JWT_TOKEN>"FEEDS={'pn://house.json':{"format":"json"},}For Moralis:MS_API_KEY="<MS_API_KEY>"FEEDS={'ms://house.json':{"format":"json"},}For Filebase or other s3 compatible servicesThe S3 pipeline requires botocore so install it.pipinstallbotocoreS3_ACCESS_KEY_ID="<S3_ACCESS_KEY_ID>"S3_SECRET_ACCESS_KEY="<S3_SECRET_ACCESS_KEY>"S3_ENDPOINT_URL="https://s3.filebase.com"S3_IPFS_URL_FORMAT="https://ipfs.filebase.io/ipfs/{cid}"FEEDS={"s3://bucket-name/foldername/%(name)s_%(time)s.json":{"format":"json"},"s3://bucket-name/foldername/%(name)s_%(time)s.csv":{"format":"csv"},}See more on FEEDShereNow perform the scrapping as you would normally.Author👤Pawan PaudelGithub:@pawanpaudel93🤝 ContributingContributions, issues and feature requests are welcome!Feel free to checkissues page.Show your supportGive a ⭐️ if this project helped you!Copyright © 2022Pawan Paudel.