Package: scrapy-pipeline-mongodb
OverviewScrapy is a great framework for web crawling. This package provides two
pipelines of saving items into MongoDB in both async and sync ways for scrapy.
Also it provides a highly customized way to interact with MongoDB in both async
and sync ways:Save an item and get Object ID with this pipelineUpdate an item and get Object ID with this pipelineRequirementsTxmongo, a async MongoDB driver with TwistedTests on Python 3.5Tests on Linux, but it’s a pure python module, should work on other platforms
with official python and Twisted supportedInstallationThe quick way:pip install scrapy-pipeline-mongodbOr put this middleware just beside the scrapy project.DocumentationSet Block Inspector inITEMPIPELINESinsettings.py, for example:from txmongo.filter import ASCENDING
from txmongo.filter import DESCENDING

# -----------------------------------------------------------------------------
# PIPELINE MONGODB ASYNC
# -----------------------------------------------------------------------------

ITEM_PIPELINES.update({
    'scrapy_pipeline_mongodb.pipelines.mongodb_async.PipelineMongoDBAsync': 500,
})

MONGODB_USERNAME = 'user'
MONGODB_PASSWORD = 'password'
MONGODB_HOST = 'localhost'
MONGODB_PORT = 27017
MONGODB_DATABASE = 'test_mongodb_async_db'
MONGODB_COLLECTION = 'test_mongodb_async_coll'

# MONGODB_OPTIONS_ = 'MONGODB_OPTIONS_'

MONGODB_INDEXES = [('field_0', ASCENDING, {'unique': True}),
                   (('field_0', 'field_1'), ASCENDING),
                   (('field_0', ASCENDING), ('field_0', DESCENDING))]

MONGODB_PROCESS_ITEM = 'scrapy_pipeline_mongodb.utils.process_item.process_item'Settings ReferenceMONGODB_USERNAMEA string of the username of the database.MONGODB_PASSWORDA string of the password of the database.MONGODB_HOSTA string of the ip address or the domain of the database.MONGODB_PORTA int of the port of the database.MONGODB_DATABASEA string of the name of the database.MONGODB_COLLECTIONA list of the indexes to create on the collection.MONGODB_OPTIONS_*Options can be attached when the pipeline start to connect to MongoBD.If any options are needed, the option can be set with the prefixMONGODB_OPTIONS_, the pipeline will parse it.For example:option nameinsettings.pyauthMechanismMONGODB_OPTIONS_authMechanismFor more options, please refer to the page:Connection String URI Format — MongoDB Manual 3.4MONGODB_INDEXESA list of the indexes defined in this setting will be created when the spider is
open.If the index has already existed, the warning or error will be suspended.MONGODB_PROCESS_ITEMThis pipeline provides a setting to define the functionprocess_item, which
can help to customize the way to interact with MongoDB. With this package there
is one default function provided: calling the methodinsert_oneofcollectionto save the item into MongoDB, then return the item.If a customize method is provided to replace the default one, please note the
behavior should follow the requirement which is clearly written in the scrapy
documents:Item Pipeline — Scrapy 1.4.0 documentationBuilt-in Functions For Processing Itemscrapy_pipeline_mongodb.utils.process_item.process_itemThis is a built-in function to call the methodinsert_oneofcollection,
and return the item.To use this function, insettings.py:MONGODB_PROCESS_ITEM = 'scrapy_pipeline_mongodb.utils.process_item.process_item'NOTEThe database drivers may have different api for the same operation, this
pipeline adopts txmongo as the async driver for MongoDB. Please read the
relative documents to make sure the customized method can run fluently in this
pipeline.Welcome to TxMongo’s documentation!pymongo – Python driver for MongoDBTODOAdd a unit test for the index created functionAdd a sync pipeline