scrapinghub/scrapylib — pull request #51 by gatufo (2014-12-23 14:22:10+00:00)
Title: Added streamitem (item, loader, exporter, storage and tests)

Added support to work with [streamcorpus](https://github.com/trec-kba/streamcorpus) Stream Items.

**Included:**
- ``streamitem.items.StreamItem``: Scrapy Item definition.
- ``streamitem.loaders.StreamItemLoader``: Scrapy Itemloader for ``StreamItem``.
- ``streamitem.exporters.StreamItemExporter``: Scrapy ItemExporter to .sc file.
- ``streamitem.storages.StreamItemFileFeedStorage``:: Scrapy FileFeedStorage to handle .sc files.

**Example:**

From Spider:

def parse_page(self, response):
loader = StreamItemLoader(item=StreamItem(), response=response)
return loader.load_item()

Export settings:

FEED_URI = ".scrapy/exports/streamitems.sc"
FEED_FORMAT = "streamcorpus"
FEED_EXPORTERS = {
'streamcorpus': 'scrapylib.streamitem.exporters.StreamItemExporter',
}
FEED_STORAGES = {
'': 'scrapylib.streamitem.storages.StreamItemFileFeedStorage',
}