Package: scrapy-toolbox
scrapy-toolboxA Python library that extends Scrapy with the following features:Error Saving to the Database Table "__errors" for manual error analysis (incl. traceback and response) and automated request reconstruction containing the following columns:failed_atspidertracebackurl (original url)request_methodrequest_urlrequest_meta (json dump that can be loaded with json.loads())request_cookies (json dump that can be loaded with json.loads())request_headers (json dump that can be loaded with json.loads())request_bodyresponse_statusresponse_urlresponse_headers (json dump that can be loaded with json.loads())response_bodyError Processing with request reconstructionDatabasePipeline for SQLAlchemyMapper to automaticaly map scrapy.Item on a database-objectMail Notification when an Exception occurs (HTTP Errors (404, 502, ...) are excluded and only stored in the Database)Automatic GitHub Issue creation when an Exception occurs (HTTP Errors (404, 502, ...) are excluded and only stored in the Database)Requisites:Environment variable "PRODUCTION" for Produciton Mode for instance in your DockerfileThe ErrorSavingMiddleware defines an errback Callback for your Requests. If you want to make use of this Feature do not define any errback.Installationpip install --upgrade scrapy-toolboxExample ProjectYou can find an example projecthere.SetupAdd the scrapy_toolbox Middlewares to your Scrapy Projectsettings.pyand set your DATABASE_DEV and DATABASE.# settings.py
SPIDER_MIDDLEWARES = {
    'scrapy_toolbox.database.DatabasePipeline': 999,
    'scrapy_toolbox.error_handling.ErrorSavingMiddleware': 1000,
    'scrapy_toolbox.error_processing.ErrorProcessingMiddleware': 1000,
}

# Example when using a MySQL
DATABASE = {
  'drivername': 'mysql+pymysql', 
  'username': '...',
  'password': '...',
  'database': '...',
  'host': '...',
  'port': '3306'
}

DATABASE_DEV = {
    'drivername': 'mysql+pymysql',
    'username': '...',
    'password': '...',
    'database': '...',
    'host': '127.0.0.1',
    'port': '3306'
}

CREATE_GITHUB_ISSUE = True # Toggle GitHub Issue creation
GITHUB_TOKEN = "..."
GITHUB_REPO = "janwendt/scrapy-toolbox" # for instance

SEND_MAILS = True # Toggle Mail Notification
MAIL_HOST = "..."
MAIL_FROM = "..."
MAIL_TO = "..."UsageSpider (Import ErrorCatcher first!!!):from scrapy_toolbox.error_handling import ErrorCatcher
import scrapy
...

class XyzSpider(scrapy.Spider, metaclass=ErrorCatcher):
...Database Pipeline:# pipelines.py
from scrapy_toolbox.database import DatabasePipeline
import xy.items as items
import xy.model as model

class ScraperXYZPipeline(DatabasePipeline):
      def __init__(self, settings):
        super().__init__(settings, items, model)# models.py
import scrapy_toolbox.database as db

# then use db.DeclarativeBase as your declarative base
class Car(db.DeclarativeBase):
  ...Query Data:# spiderXYZ.py
session = self.crawler.database_session
session.query(models.Market.id, models.Market.zip_code).all()Process Errors:scrapy crawl spider_xyz -a process_errors=TrueLimitationsSyntax Errors in your settings.py are not handled.Supported versionsThis package works with Python 3. It has been tested with Scrapy up to version 1.4.0.Tasklist[] Error Processing[] Scaffold for instance ItemPipelineBuild Realeasepython setup.py sdist bdist_wheel
cd dist
pip install --upgrade --no-deps --force-reinstall scrapy_toolbox-0.3.3-py3-none-any.whl
cd ..
twine upload dist/*