Package: scrapy-s3pipeline
Scrapy S3 PipelineScrapy pipeline to store items intoAmazon S3orGoogle Cloud Storage (GCS)bucket. Unlike built-inFeedExporter, the pipeline has the following features:The pipeline upload items to S3/GCS by chunk while crawler is running.From Scrapy 2.3, built-inFEED_EXPORT_BATCH_ITEM_COUNTdoes almost the same thing.Support GZip compression.The pipeline aims to run crawler and scraper in different processes, e.g. run crawler process with Scrapy in AWS Fargate and run scraper process with lxml in AWS Lambda.RequirementsPython 3.6+ (Tested in 3.9)Scrapy 1.1+ (Tested in 2.4)boto3 or google-cloud-storageInstallFor S3 users:$pip3installscrapy-s3pipeline[s3]For GCS users:$pip3installscrapy-s3pipeline[gcs]Getting startedInstall Scrapy S3 Pipeline with pip.$pip3installscrapy-s3pipeline[s3]or$pip3installscrapy-s3pipeline[gcs]Add's3pipeline.S3Pipeline'toITEM_PIPELINESsetting in your Scrapy project.ITEM_PIPELINES={'s3pipeline.S3Pipeline':100,# Add this line.}AddS3PIPELINE_URLsetting. You need to changemy-bucketto your bucket name.# For S3 usersS3PIPELINE_URL='s3://my-bucket/{name}/{time}/items.{chunk:07d}.jl.gz'# For GCS usersS3PIPELINE_URL='gs://my-bucket/{name}/{time}/items.{chunk:07d}.jl.gz'GCS_PROJECT_ID='my-project'# Change to your project idSetup AWS/GCP credentials.For S3 users:Setup AWS credentials viaaws configurecommand orenvironment variables. Alternatively, use Scrapy's settingsAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY.For GCS users:Setup GCP credentials viagcloud auth application-default logincommand or environment variableGOOGLE_APPLICATION_CREDENTIALS. Alternatively, you can set json string of service account's key file toGOOGLE_APPLICATION_CREDENTIALS_JSONsettings.Run your spider. You will see items in your bucket after 100 items are crawled or the spider is closed.SettingsS3PIPELINE_URL (Required)S3/GCS Bucket URL to store items.e.g.:S3:s3://my-bucket/{name}/{time}/items.{chunk:07d}.jl.gzGCS:gs://my-bucket/{name}/{time}/items.{chunk:07d}.jl.gzThe following replacement fields are supported inS3PIPELINE_URL.{chunk}- gets replaced by a start index of items in current chunk, e.g. '0', '100', '200',....{time}- gets replaced by a timestamp when the spider is started.You can also use other spider fields, e.g.{name}. You can useformat string syntaxhere, e.g.{chunk:07d}.File format is determined by a file extension in the URL. For example, ifS3PIPELINE_URLends with.jsonor.json.gz, JSON format is used. See Scrapy's built-inFEED_EXPORTERSsettings for supported formats. If the file extension is not available inFEED_EXPORTERS, JSONLines format is used by default.S3PIPELINE_MAX_CHUNK_SIZE (Optional)Default:100Max count of items in a single chunk.S3PIPELINE_MAX_WAIT_UPLOAD_TIME (Optional)Default:30.0When no new item is processed in more thanS3PIPELINE_MAX_WAIT_UPLOAD_TIMEseconds, it will be forced to upload a chunk.S3PIPELINE_GZIP (Optional)Default:TrueifS3PIPELINE_URLends with.gz; otherwiseFalse.IfTrue, uploaded files will be compressed with Gzip.Page itemFor convinience, Scrapy S3 Pipeline providess3pipeline.Pageitem class to store entire HTTP body. It hasurl,bodyandcrawled_atfields.This make it easy to store entire HTTP body and run scraper in other process. It's friendly to server-less architecture which run scraper in AWS Lambda.Example usage of Page:fromdatetimeimportdatetime,timezoneimportscrapyfroms3pipelineimportPage# ...classYourSpider(scrapy.Spider):# ...defparse(self,response):# You can create Page instance just one line.yieldPage.from_response(response)# Or, you can fill item fields manually.item=Page()item['url']=response.urlitem['body']=response.textitem['crawled_at']=datetime.now(timezone.utc).replace(microsecond=0).isoformat()yielditemNote: Page's body is omitted when printed to logs to improve readbility of logs.DevelopmentTest$ python3 setup.py testRelease$ pip install twine wheel$ python3 setup.py bdist_wheel sdist
$ twine upload dist/*