scrapinghub/scrapylib — pull request #50 by gatufo (2014-12-22 13:38:49+00:00)
Title: Added chunkexports extension

An extension to break item exports into chunks.

**Settings**:
- **CHUNKED_FEED_URI**: The feed uri to use for exporting (Overrides FEED_URI setting).
- **CHUNKED_FEED_FORMAT**: The feed format to use for exporting (Overrides FEED_FORMAT setting).
- **CHUNKED_FEED_ITEMS_PER_CHUNK**: Number of items included in each chunk.
- **CHUNKED_FEED_TIMESTAMP_FORMAT**: The format to be used for the timestamp uri parameter.

**Example**:

CHUNKED_FEED_URI = 'export_%(chunk_number)02d.json'
CHUNKED_FEED_FORMAT = 'json'
CHUNKED_FEED_ITEMS_PER_CHUNK = 100

For 250 items will generate the following files:
- export_01.json (100 items)
- export_02.json (100 items)
- export_03.json (50 items)

**Available uri format values**:
- **chunk_number**: The active chunk counter. (Starts in 1).
- **scrapy_job**: The Scrapy job (if available).
- **scrapy_project_id**: The Scrapy job id (if available).
- **timestamp**: Current timestamp in UTC (formatted with CHUNKED_FEED_TIMESTAMP_FORMAT setting).