Saravanan666/pypi-org huggingface dataset:
Package: scrapy-httpcache
Overviewscrapy-httpcache is a scrapy middleware to save http cache in mongodb.
Besides, scrapy-httpcache contains two extra storage plugin,
including request_error_storage and banned_storage.
request_error_storage can save Request which occur error.
banned_storage can save Banned Request whose block_checker can be override.RequirementsPython 3.3+Works on Linux, Windows, Mac OSX, BSDInstallThe quick way:pip install scrapy-httpcacheOR copy this middleware to your scrapy project.DocumentationIn settings.py, for example:# -----------------------------------------------------------------------------
# SCRAPY HTTPCACHE SETTINGS
# -----------------------------------------------------------------------------
DOWNLOADER_MIDDLEWARES.update({
    'scrapy.downloadermiddlewares.httpcache.HttpCacheMiddleware': None,
    'scrapy_httpcache.downloadermiddlewares.httpcache.AsyncHttpCacheMiddleware': 900,
})

HTTPCACHE_ENABLED = True
HTTPCACHE_IGNORE_HTTP_CODES = [301, 302, 500, 503]
HTTPCACHE_STORAGE = 'scrapy_httpcache.extensions.httpcache_storage.MongoDBCacheStorage'
HTTPCACHE_MONGODB_HOST = '127.0.0.1'
HTTPCACHE_MONGODB_PORT = 27017
HTTPCACHE_MONGODB_USERNAME = 'root'
HTTPCACHE_MONGODB_PASSWORD = 'password'
HTTPCACHE_MONGODB_CONNECTION_POOL_KWARGS = {}
HTTPCACHE_MONGODB_AUTH_DB = 'admin'
HTTPCACHE_MONGODB_DB = 'cache_storage'
HTTPCACHE_MONGODB_COLL = 'cache'

# -----------------------------------------------------------------------------
# SCRAPY HTTPCACHE BANNED SETTINGS (optional)
# -----------------------------------------------------------------------------
BANNED_STORAGE = 'scrapy_httpcache.extensions.banned_storage.MongoBannedStorage'

# -----------------------------------------------------------------------------
# SCRAPY HTTPCACHE REQUEST ERROR SETTINGS (optional)
# -----------------------------------------------------------------------------
REQUEST_ERROR_STORAGE = 'scrapy_httpcache.extensions.request_error_storage.MongoRequestErrorStorage'If you want to remove banned response, usesend_catch_log_deferredto send signal toscrapy_httpcache.signals.remove_bannedwith kwargs contains (spider, response, exception), which callback function return a Deferred.
Package: scrapy-http-pipeline
scrapy-http-pipelineJust a simple Scrapy HTTP pipeline to POST your items to your server.UsageInstallpip install scrapy-http-pipelineConfigure scrapy settings.pyITEM_PIPELINES = {
    'scrapyhttppipeline.scrapyhttppipeline.HttpPostPipeline': 500
}

# Url to your server, which accepts POST requests
HTTP_POST_PIPELINE_URL = 'http://localhost:8080/items'

# Any custom headers you want to add, e.g. authentication
HTTP_POST_PIPELINE_HEADERS = {
    'X-Authorization': 'xxx'
}

# If you want to send more items at once (and have less HTTP POST requests incoming.)
# If True items will be send as [{key1:val1},{key1:val1}] instead of {key1:val1}
HTTP_POST_PIPELINE_BUFFERED = False
HTTP_POST_PIPELINE_BUFFER_SIZE = 100DevelopingPackage requirements are handled using pip. To install them dopip install -r requirements.txtTestsTesting is set up usingpytestand coverage is handled
with the pytest-cov plugin.Run your tests withpy.testin the root directory.Coverage is ran by default and is set in thepytest.inifile.
To see an html output of coverage openhtmlcov/index.htmlafter running the tests.Travis CIThere is a.travis.ymlfile that is set up to run your tests for python 2.7
and python 3.3-3.7, should you choose to use it.
Package: scrapy_huo_utilities
scrapy_huo_utilities介绍包含一个随机UserAgent和基于leveldb的DuplicateUrls中间件、几个清洗html的函数,自用的几个常用功能。安装教程pip install scrapy_huo_utilities使用说明RandomUserAgent_DOWNLOADER_MIDDLEWARES:在settings.py的DOWNLOADER_MIDDLEWARES中修改添加："scrapy_huo_utilities.MIDDLEWARES.Downloader_Middleware_Utils.RandomUserAgent_DOWNLOADER_MIDDLEWARES": 543,DuplicateUrls_SPIDER_MIDDLEWARES:
在settings.py的SPIDER_MIDDLEWARES中修改添加："scrapy_huo_utilities.MIDDLEWARES.Spider_Middleware_Utils.DuplicateUrls_SPIDER_MIDDLEWARES": 543,html_clean:from scrapy_huo_utilities.PROCESSORS import clean_html_attributes,clean_html_tags,clean_empty_img_tagsclean_empty_img_tags:清除没有src属性或src属性为空字符串的img标签。clean_empty_a_tags:清除没有href属性或href属性为空字符串的a标签。clean_html_tags:用于清除指定列表中的HTML标签。:param html_data:  传入html
:param remove_tags:删除标签列表
:param reserve_content:是否保留删除标签的文本内容，默认保留。
:return: 处理后的htmlclean_html_attributes：用于清除HTML标签中的属性，除了白名单中包含的属性。:param html_data:传入html
:param whitelist:白名单
:return: html_data处理后的html
Package: scrapy-impersonate
scrapy-impersonatescrapy-impersonateis a Scrapy download handler. This project integratescurl_cffito perform HTTP requests, so it can impersonate browsers' TLS signatures or JA3 fingerprints.Installationpip install scrapy-impersonateActivationReplace the defaulthttpand/orhttpsDownload Handlers throughDOWNLOAD_HANDLERSDOWNLOAD_HANDLERS={"http":"scrapy_impersonate.ImpersonateDownloadHandler","https":"scrapy_impersonate.ImpersonateDownloadHandler",}Also, be sure toinstall the asyncio-based Twisted reactor:TWISTED_REACTOR="twisted.internet.asyncioreactor.AsyncioSelectorReactor"Basic usageSet theimpersonateRequest.metakey to download a request usingcurl_cffi:importscrapyclassImpersonateSpider(scrapy.Spider):name="impersonate_spider"custom_settings={"DOWNLOAD_HANDLERS":{"http":"scrapy_impersonate.ImpersonateDownloadHandler","https":"scrapy_impersonate.ImpersonateDownloadHandler",},"TWISTED_REACTOR":"twisted.internet.asyncioreactor.AsyncioSelectorReactor",}defstart_requests(self):forbrowserin["chrome110","edge99","safari15_5"]:yieldscrapy.Request("https://tls.browserleaks.com/json",dont_filter=True,meta={"impersonate":browser},)defparse(self,response):# ja3_hash: 773906b0efdefa24a7f2b8eb6985bf37# ja3_hash: cd08e31494f9531f560d64c695473da9# ja3_hash: 2fe1311860bc318fc7f9196556a2a6b9yield{"ja3_hash":response.json()["ja3_hash"]}Supported browsersThe following browsers can be impersonatedBrowserVersionBuildOSName9999.0.4844.51Windows 10chrome999999.0.4844.73Android 12chrome99_android100100.0.4896.75Windows 10chrome100101101.0.4951.67Windows 10chrome101104104.0.5112.81Windows 10chrome104107107.0.5304.107Windows 10chrome107110110.0.5481.177Windows 10chrome110116116.0.5845.180Windows 10chrome116119119.0.6045.199macOS Sonomachrome119120120.0.6099.109macOS Sonomachrome1209999.0.1150.30Windows 10edge99101101.0.1210.47Windows 10edge10115.316612.4.9.1.8MacOS Big Sursafari15_315.517613.2.7.1.8MacOS Montereysafari15_517.0unclearMacOS Sonomasafari17_017.2uncleariOS 17.2safari17_2_iosThanksThis project is inspired by the following projects:curl_cffi- Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.curl-impersonate- A special build of curl that can impersonate Chrome & Firefoxscrapy-playwright- Playwright integration for Scrapy
Package: scrapy-inline-requests
Scrapy Inline RequestsA decorator for writing coroutine-like spider callbacks.Free software: MIT licenseDocumentation:https://scrapy-inline-requests.readthedocs.org.Python versions: 2.7, 3.4+QuickstartThe spider below shows a simple use case of scraping a page and following a few links:frominline_requestsimportinline_requestsfromscrapyimportSpider,RequestclassMySpider(Spider):name='myspider'start_urls=['http://httpbin.org/html']@inline_requestsdefparse(self,response):urls=[response.url]foriinrange(10):next_url=response.urljoin('?page=%d'%i)try:next_resp=yieldRequest(next_url)urls.append(next_resp.url)exceptException:self.logger.info("Failed request%s",i,exc_info=True)yield{'urls':urls}See theexamples/directory for a more complex spider.WarningThe generator resumes its execution when a request’s response is processed,
this means the generator won’t be resume after yielding an item or a request
with it’s own callback.Known IssuesMiddlewares can drop or ignore non-200 status responses causing the callback
to not continue its execution. This can be overcome by using the flaghandle_httpstatus_all. See thehttperror middlewaredocumentation.High concurrency and large responses can cause higher memory usage.This decorator assumes your method have the following signature(self, response).Wrapped requests may not be able to be serialized by persistent backends.Unless you know what you are doing, the decorated method must be a spider
method and return ageneratorinstance.History0.3.1 (2016-07-04)Added deprecation about decorating non-spider functions.Warn if the callback returns requests with callback or errback set. This
reverts the compability with requests with callbacks.0.3.0 (2016-06-24)~~Backward incompatible change: Added more restrictions to the request object (no callback/errback).~~Cleanup callback/errback attributes before sending back the request to the
generator. This fixes an edge case when usingrequest.replace().Simplified example spider.0.2.0 (2016-06-23)Python 3 support.0.1.2 (2016-05-22)Scrapy API and documentation updates.0.1.1 (2013-02-03)Minor tweaks and fixes.0.1.0 (2012-02-03)First release on PyPI.
Package: scrapy-item
scrapy-itemItem with general/unknown/dynamic fields.
Variation ofhttps://github.com/WilliamKinaan/ScrapyDynamicItems- extracting only essential part.
See [https://medium.com/@alex_ber/scrapy-item-with-general-fields-7552bd6e4622] for description.
See CHANGELOG.md for detail description.Getting HelpQuickStartpython3-mpipinstall-Uscrapy-itemInstalling from Githubpython3-mpipinstall-Uhttps://github.com/alex-ber/scrapy-item/archive/master.zipOptionally installing tests requirements.python3-mpipinstall-Uhttps://github.com/alex-ber/scrapy-item/archive/master.zip#egg=alex-ber-utils[tests]Or explicitly:wgethttps://github.com/alex-ber/scrapy-item/archive/master.zip-Omaster.zip;unzipmaster.zip;rmmaster.zipAnd then installing from source (see below).Installing from sourcepython3-mpipinstall-rreq.txt# only installs "required" (relaxed)python3-mpipinstall.# only installs "required"python3-mpipinstall.[tests]# installs dependencies for testsAlternatively you install install from requirements file:python3-mpipinstall-rrequirements.txt# only installs "required"python3-mpipinstall-rrequirements-tests.txt# installs dependencies for testsFrom the directory with setup.pypython3setup.pytest#run all testsorpytestInstalling new versionSeehttps://docs.python.org/3.1/distutils/uploading.htmlpython3setup.pysdistuploadRequirementsscrapy-item requires the following modules.Python 3.6+ChangelogItem with general/unknown/dynamic fields.
Variation ofhttps://github.com/WilliamKinaan/ScrapyDynamicItems- extracting only essential part.
See [https://medium.com/@alex_ber/scrapy-item-with-general-fields-7552bd6e4622] for description.All notable changes to this project will be documented in this file.#https://pypi.org/manage/project/scrapy-state/releases/[Unrelased][0.0.2] - 01/07/2020AddedUnit-test forGeneralItemclass inalexber.scrapyitemmodule is added.[0.0.1] - 01/07/2020AddedGeneralItemclass inalexber.scrapyitemmodule. Item with general/unknown/dynamic fields.
Variation of [https://github.com/WilliamKinaan/ScrapyDynamicItems] - extracting only essential part.See [https://medium.com/@alex_ber/scrapy-item-with-general-fields-7552bd6e4622] for description.
Package: scrapy-itemagic
# scrapy-itemagic
Item parsing tools for Scrapy[![Latest Version](https://pypip.in/version/scrapy-itemagic/badge.svg)](https://pypi.python.org/pypi/scrapy-itemagic/)
[![Downloads](https://pypip.in/download/scrapy-itemagic/badge.svg)](https://pypi.python.org/pypi/scrapy-itemagic/)
[![Development Status](https://pypip.in/status/scrapy-itemagic/badge.svg)](https://pypi.python.org/pypi/scrapy-itemagic/)## Examples
[TODO]## Installation
Viapip:`shell $ pip installscrapy-itemagic`or from the source :`shell $ git clonegit@github.com:as0n/scrapy-itemagic.git$ cdscrapy-itemagic$ python setup.py install `## API
[TODO]
Package: scrapy-item-pipelines
scrapy-item-pipelinesVarious scrapy item pipelinesInstallationpip install scrapy-item-pipelinesSaveToKafkaPipelineItem pipeline to push items to kafka. Items will be converted into JSON format and pushed to a defined kafka topic.SettingsSL_SCRAPY_ITEM_PIPELINES_SETTINGS = {
    "push_to_kafka_hosts": "localhost:9092"  # Kafka broker hosts. Separated with a comma.
    "push_to_kafka_default_topic": ""  # kafka default topic.
}UsageIf items should be pushed to different kafka topics per item, the topic can be defined in the item class.
Also if a data key should be pushed to kafka we can define the item field value to use by defining it
in the item class. If nokafka_data_keyis defined no data key will be pushed.class DemoItem(scrapy.Item):
    kafka_topic = "topic-to-push-items"
    kafka_data_key = "another_unique_field"

    field_name = scrapy.Field()
    another_unique_field = scrapy.Field()After configuring addscrapy_item_pipelines.streaming.PushToKafkaPipelineto the ITEM_PIPELINES setting.ITEM_PIPELINES = {
    ...
    ...
    "scrapy_item_pipelines.streaming.PushToKafkaPipeline": 999,
}FilterDuplicatesPipelineItem pipeline to filter out duplicate items calculated using defined keys in the item.UsageDefine an attribute called unique_key in the item. If the unique key is a single field
unique_key can be defined as a string or if the unique key is a multi field key unique_key
should be a tuple of strings. If no unique_key is defined filtering will be done usingidfield.
If you want to skip duplicate filtering for an item define unique_key as None.The pipeline will include a stats calledduplicate_item_countwhich is the number
of duplicate items dropped.class DemoItem(scrapy.Item):
    field1 = scrapy.Field()
    field2 = scrapy.Field()

    unique_key = None  # duplicates won't be filtered.


class DemoItem(scrapy.Item):
    # No unique_key is defined. Filtering will be done using `id` field.
    field1 = scrapy.Field()
    field2 = scrapy.Field()
    id = scrapy.Field()


class DemoItem(scrapy.Item):
    field1 = scrapy.Field()
    field2 = scrapy.Field()

    unique_key = "field1"  # Duplicates will be filtered using field1.


class DemoItem(scrapy.Item):
    field1 = scrapy.Field()
    field2 = scrapy.Field()

    unique_key = ("field1", "field2")  # Duplicates will be filtered using both field1 and field2Addscrapy_item_pipelines.misc.FilterDuplicatesPipelineto the ITEM_PIPELINES setting.ITEM_PIPELINES = {
    ...
    ...
    "scrapy_item_pipelines.misc.FilterDuplicatesPipeline": 500,
}
Package: scrapy-job-parameters
# Scrapy Metas Extension[![CircleCI](https://circleci.com/gh/Kpler/scrapy-job-parameters-extension.svg?style=svg)](https://circleci.com/gh/Kpler/scrapy-job-parameters-extension)> Scrapy extension to make env meta information available as spider fields.Current implementation exposes to the spider ametaobject with the
following attributes:project_id- as defined by Scrapinghubspider_id-job_id-job_name- raw job id from Scrapinghub or uuid v4job_time- excecution time
Package: scrapy-jsonrpc
scrapy-jsonrpc is an extension to control a running Scrapy web crawler via
JSON-RPC. The service provides access to the main Crawler object via theJSON-RPC 2.0protocol.It’s enabled by theJSONRPC_ENABLEDsetting. The web server will listen
in the port specified inJSONRPC_PORT, and will log to the file
specified inJSONRPC_LOGFILE.The endpoint for accessing the crawler object is:http://localhost:6080/crawlerExample clientThere is a command line tool provided for illustration purposes on how to build
a client. You can find it inexample-client.py. It supports a few basic
commands such as listing the running spiders, etc.SettingsThese are the settings that control the web service behaviour:JSONRPC_ENABLEDDefault:TrueA boolean which specifies if the web service will be enabled (provided its
extension is also enabled).JSONRPC_LOGFILEDefault:NoneA file to use for logging HTTP requests made to the web service. If unset web
the log is sent to standard scrapy log.JSONRPC_PORTDefault:[6080, 7030]The port range to use for the web service. If set toNoneor0, a
dynamically assigned port is used.JSONRPC_HOSTDefault:'127.0.0.1'The interface the web service should listen on.
Package: scrapy-kafka
Kafka-based components for Scrapy, to allowSpider`s to read their next URL
from a Kafka topic and an `ItemPipelineto store the JSON-ifiedItemback
into a Kafka topic.Please see theexampledirectory for how to use this.ContributorsContributors toscrapy-kafka, listed alphabetically:Matthew Daniel@mdanielDidier Deshommes@dfdeshom
Package: scrapy-kit
scrapy-kitA library for scrapy tools, including but not limited to the usual pipelines, middlewares, etc.installpipinstallscrapy-kitusagepipelinesmongo pipelineA pipeline saved into MongoDB asynchronously with txmongouse database
db.createUser(
{
user: "username",
pwd: "password",
roles: [ { role: "readWrite", db: "database" } ]
}
)how use:add to settings.pyITEM_PIPELINES={'scrapy_kit.pipelines.MongoPipeline':300,}add mongo config to settings.pyMONGO_URI="mongodb://username:password@host:port"MONGO_DATABASE_NAME="database"MONGO_COLLECTION_NAME="collection"middlewaresproxy connection close middlewareProxy close connection multiplexing middlewareTunnel Proxy Dynamic Edition request found that the number of requests in the Personal Centre Tunnel Proxy Usage
Statistics is very small, which is seriously inconsistent with the real number of requests.
Moreover, there is no IP change when using Tunnel Broker Dynamic Edition.
The reason for this is that the tunnel sends requests that reuse previously established connections.
You need to add Connection: close to the header.How to use it:Add in settings.py:DOWNLOADER_MIDDLEWARES={'scrapy_kit.middlewares.ProxyConnectionCloseMiddleware':543,}developmentgitclonecdscrapy-kit
poetryinstall
Package: scrapylib
This library is deprecated and unmaintained.Some of its components were moved to their own package:Old locationNew locationscrapylib.crawlerascrapy-crawlerascrapylib.deltafetchscrapy-deltafetchscrapylib.hcfscrapy-hcfscrapylib.magicfieldsscrapy-magicfieldsscrapylib.querycleanerscrapy-querycleanerscrapylib.splitvariantsscrapy-splitvariants
Package: scrapy-link-filter
Scrapy-link-filterSpider Middleware that allows aScrapy Spiderto filter requests.
There is similar functionality in theCrawlSpideralready using Rules and in theRobotsTxtMiddleware, but there are twists.
This middleware allows defining rules dinamically per request, or as spider arguments instead of project settings.InstallThis project requiresPython 3.6+andpip. Using avirtual environmentis strongly encouraged.$pipinstallgit+https://github.com/croqaz/scrapy-link-filterUsageFor the middleware to be enabled as a Spider Middleware, it must be added in the projectsettings.py:SPIDER_MIDDLEWARES = {
    # maybe other Spider Middlewares ...
    # can go after DepthMiddleware: 900
    'scrapy_link_filter.middleware.LinkFilterMiddleware': 950,
}Or, it can be enabled as a Downloader Middleware, in the projectsettings.py:DOWNLOADER_MIDDLEWARES = {
    # maybe other Downloader Middlewares ...
    # can go before RobotsTxtMiddleware: 100
    'scrapy_link_filter.middleware.LinkFilterMiddleware': 50,
}The rules must be defined either in the spider instance, in aspider.extract_rulesdict, or per request, inrequest.meta['extract_rules'].
Internally, the extract_rules dict is converted into aLinkExtractor, which is used to match the requests.Notethat the URL matching is case-sensitive by default, which works in most cases. To enable case-insensitive matching, you can specify a "(?i)" inline flag in the beggining of each "allow", or "deny" rule that needs to be case-insensitive.Example of a specific allow filter, on a spider instance:fromscrapy.spidersimportSpiderclassMySpider(Spider):extract_rules={"allow_domains":"example.com","allow":"/en/items/"}Or a specific deny filter, inside a request meta:request.meta['extract_rules']={"deny_domains":["whatever.com","ignore.me"],"deny":["/privacy-policy/?$","/about-?(us)?$"]}The possible fields are:allow_domainsanddeny_domains- one, or more domains to specifically limit to, or specifically rejectallowanddeny- one, or more sub-strings, or patterns to specifically allow, or rejectAll fields can be defined as string, list, set, or tuple.LicenseBSD3© Cristi Constantin.
Package: scrapy_loaders
scrapy_loadersScrapy Pipelines LoadersFree software: MIT licenseInstallpipinstallscrapy_loadersAt Your Scrapy project (Example: SpiderProject)models.pyfromsqlalchemyimport(Column,String,Text,)fromsqlalchemy.ext.declarativeimportdeclarative_baseDeclarativeBase=declarative_base()classItemModel(DeclarativeBase):__tablename__='table_name'id=Column('id',String(10),primary_key=True)name=Column('name',String(60))description=Column('description',Text())url=Column('url',Text())md5sum=Column('md5sum',String(45))......settings.py...# Postgres settings, check other SQLAlchemy settings if you wishDATABASE={'drivername':'postgresql+psycopg2','host':'localhost','port':'5432','username':'username','password':'password','database':'attack_mitre',}DECLARATIVE_BASE='SpiderProject.models.DeclarativeBase'...ITEM_PIPELINES={'SpiderProject.pipelines.SpiderProjectDbPipeline':300,}...pipelines.pyfromSpiderProject.modelsimportItemModelfromscrapy_loaders.db_loadersimportDBLoaderfromscrapy_loaders.pipelinesimportDbPipelineclassItemLoader(DBLoader):model=ItemModelhash_fields=['name','description']update_fields=hash_fields+['md5sum']...classSpiderProjectDbPipeline(DbPipeline):db_loaders={'Item':ItemLoader,}...FeaturesTestsTODO: testsnosetests--with-coverage--cover-inclusive--cover-package=scrapy_loaders--cover-html
Package: scrapy-mailgun
Permission is hereby granted, free of charge, to any person obtaining a copyof this software and associated documentation files (the "Software"), to dealin the Software without restriction, including without limitation the rightsto use, copy, modify, merge, publish, distribute, sublicense, and/or sellcopies of the Software, and to permit persons to whom the Software isfurnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in allcopies or substantial portions of the Software.THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS ORIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THEAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHERLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THESOFTWARE.Description: # Scrapy-mailgun: Mailgun integration for ScrapyA [mailgun](https://mailgun.com) integration for scrapy.## Quick StartInstall via `pip`:```$ pip install scrapy-mailgun```What's available right now is an `EmailReport` extension, that allows you to send a report once a spider is done through mailgun.Configure:```pythonEXTENSIONS = {'scrapymailgun.extensions.EmailReport': True,}SM_DOMAIN = 'api.mailgun.net/v3/XXXXXXXXXXXXXXXX.mailgun.org'SM_KEY = 'key-XXXXXXXXXXXXXXXXX'SM_FROM = 'from-email@acme.org'SM_TO = 'to-email@acme.org'  # comma sparatedSM_EMAIL_REPORT_TMPL_SUBJECT = 'Your spider [{{spider.name}}] is done: {{reason}}'SM_EMAIL_REPORT_TMPL_BODY = 'job stats: {{ spider.crawler.stats._stats }}'```### TemplatesCurrently verbatim template strings are supported for flexible scraper configuration (you can change these remotely with no deploy).As for template variables, what's available to you is your `spider` and `reason` instances directly. Do what ever you want with those in the templates.## DevelopersSet up a development environment```$ pip install -r requirements.txt```### Development* Dependencies: list them in `requirements.txt`### Release* Dependencies: list them in `setup.py` under `install_requires`:```pythoninstall_requires=['peppercorn'],```Then:```$ make dist && make release```# ContributingFork, implement, add tests, pull request, get my everlasting thanks and a respectable place here :).### Thanks:To all [Contributors](https://github.com/jondot/scrapy-mailgun/graphs/contributors) - you make this happen, thanks!# CopyrightCopyright (c) 2017 [Dotan Nahum](http://gplus.to/dotan) [@jondot](http://twitter.com/jondot). See [LICENSE](LICENSE) for further details.Platform: UNKNOWN