Package: scrapy-sqlitem
scrapy-sqlitemscrapy-sqlitem allows you to define scrapy items using Sqlalchemy models
or tables. It also provides an easy way to save to the database in
chunks.This project is in beta. Pull requests and feedback are welcome. The
regular caveats of using a sql database backend for a write heavy
application still apply.Inspiration fromscrapy-redisandscrapy-djangoitemQuickstartpip install scrapy_sqlitemDefine items using Sqlalchemy ORMfromscrapy_sqlitemimportSqlItemclassMyModel(Base):__tablename__='mytable'id=Column(Integer,primary_key=True)name=Column(String)classMyItem(SqlItem):sqlmodel=MyModelOr Define Items using Sqlalchemy Corefromscrapy_sqlitemimportSqlItemclassMyItem(SqlItem):sqlmodel=Table('mytable',metadataColumn('id',Integer,primary_key=True),Column('name',String,nullable=False))If tables have not been created yet make sure to create them. See
sqlalchemy docs and the example spider.Use SqlSpider to easily save scraped items to the databasesettings.pyDATABASE_URI="sqlite:///"Define your spiderfromscrapy_sqlitemimportSqlSpiderclassMySpider(SqlSpider):name='myspider'start_urls=('http://dmoz.org',)defparse(self,response):selector=Selector(response)item=MyItem()item['name']=selector.xpath('//title[1]/text()').extract_first()yielditemRun the spiderscrapycrawlmyspiderQuery the databaseSelect*frommytable;id|name|----+-----------------------------------+1|DMOZ-theOpenDirectoryProject|Other InformationDo not want to use SqlSpider? Write a pipeline instead.fromsqlalchemyimportcreate_engineclassCommitSqlPipeline(object):def__init__(self):self.engine=create_engine("sqlite:///")defprocess_item(self,item,spider):item.commit_item(engine=self.engine)Drop items missing required primary key data before saving to the dbfromscrapy.exceptionsimportDropItemclassDropMissingDataPipeline(object):defprocess_item(self,item,spider):ifitem.null_required_fields:raiseDropItemelse:returnitem# Watch out for Serial primary keys that are considered null.Save to the database in chunks rather than item by itemInherit from SqlSpider and..In settingsDEFAULT_CHUNKSIZE=500CHUNKSIZE_BY_TABLE={'mytable':1000,'othertable':250}If an error occurs while saving a chunk to the db it will try and save
each item one at a timeAccess the underlying sqlalchemy table to query the databaseINSERTINTOmytable(id,name)VALUES('1','ryan')myitem=MyItem()# bind the table to an engine (I could have done this when I created the table too)myitem.table.metadata.bind=self.enginemyitem.table.select().where(item.table.c.id==1).execute().fetchone()(1,'ryan')What row in the database matches the data in my item?myitem=MyItem()myitem['id']=1myitem.get_matching_dbrow(bind=self.engine)(1,'ryan')This is same query as the one above!GotchasIf you subclass either item_scraped or spider_closed make sure to call super!classMySpider(SqlSpider):defparse(self,response):passdefspider_closed(self,spider,reason):super(MySpider,self).spider_closed(spider,reason)self.log("Log some really important custom stats")Be Careful with other Mixins. The inheritance structure can get a little
messy. If a class early in the mro subclasses item_scraped and does not
call super the item_scraped method of SqlSpider will never get called.Other Methods of sqlitemsqlitem.tablereturns the sqlalchemy core table that corresponds to that item.sqlitem.null_required_fieldsreturns a set of the database key names that are are marked not
nullable and the corresponding data in the item is null.sqlitem.null_primary_key_fieldsreturns a set of the primary key names where the corresponding data
in the item is null.sqlitem.primary_keyssqlitem.required_keyssqlitem.get_matching_dbrow(bind=None, use_cache=True)Find the data in the database that matches the primary key data in
the itemToDoContinuous integration Tests