Saravanan666/pypi-org huggingface dataset:
Package: scrapydweb
English |简体中文ScrapydWeb: Web app for Scrapyd cluster management, with support for Scrapy log analysis & visualization.Scrapyd ScrapydWeb LogParserRecommended ReadingHow to efficiently manage your distributed web scraping projectsHow to set up Scrapyd cluster on HerokuDemoscrapydweb.herokuapp.comFeaturesView contentsScrapyd Cluster ManagementAll Scrapyd JSON API SupportedGroup, filter and select any number of nodesExecute command on multinodes with just a few clicksScrapy Log AnalysisStats collectionProgress visualizationLogs categorizationEnhancementsAuto packagingIntegrated withLogParserTimer tasks:e-mail:Monitor & AlertMobile UIBasic auth for web UIGetting StartedView contentsPrerequisitesMake sure thatScrapydhas been installed and started on all of your hosts.Note that for remote access, you have to manually set 'bind_address = 0.0.0.0' inthe configuration file of Scrapydand restart Scrapyd to make it visible externally.InstallUse pip:pipinstallscrapydwebNote that you may need to executepython -m pip install --upgrade pipfirst in order to get the latest version of scrapydweb, or download the tar.gz file fromhttps://pypi.org/project/scrapydweb/#filesand get it installed viapip install scrapydweb-x.x.x.tar.gzUse git:pipinstall--upgradegit+https://github.com/my8100/scrapydweb.gitOr:gitclonehttps://github.com/my8100/scrapydweb.gitcdscrapydweb
pythonsetup.pyinstallStartStart ScrapydWeb via commandscrapydweb. (a config file would be generated for customizing settings at the first startup.)Visithttp://127.0.0.1:5000(It's recommended to use Google Chrome for a better experience.)Browser SupportThe latest version of Google Chrome, Firefox, and Safari.Running the testsView contents$gitclonehttps://github.com/my8100/scrapydweb.git
$cdscrapydweb# To create isolated Python environments$pipinstallvirtualenv
$virtualenvvenv/scrapydweb# Or specify your Python interpreter: $ virtualenv -p /usr/local/bin/python3.7 venv/scrapydweb$sourcevenv/scrapydweb/bin/activate# Install dependent libraries(scrapydweb)$pythonsetup.pyinstall(scrapydweb)$pipinstallpytest(scrapydweb)$pipinstallcoverage# Make sure Scrapyd has been installed and started, then update the custom_settings item in tests/conftest.py(scrapydweb)$vitests/conftest.py(scrapydweb)$curlhttp://127.0.0.1:6800# '-x': stop on first failure(scrapydweb)$coveragerun--source=scrapydweb-mpytesttests/test_a_factory.py-s-vv-x(scrapydweb)$coveragerun--source=scrapydweb-mpytesttests-s-vv--disable-warnings(scrapydweb)$coveragereport# To create an HTML report, check out htmlcov/index.html(scrapydweb)$coveragehtmlBuilt WithView contentsFront EndElementEChartsBack EndFlaskChangelogDetailed changes for each release are documented in theHISTORY.md.Authormy8100ContributorsKaislaLicenseThis project is licensed under the GNU General Public License v3.0 - see theLICENSEfile for details.
Package: scrapy-dynamodb
scrapy-dynamodb allows you to export your items to a DynamoDB database.InstallationYou can installscrapy-dynamodbeither via the Python Package Index (PyPI)
or from source.To install usingpip,:$ pip install scrapy-dynamodbTo install usingeasy_install,:$ easy_install scrapy-dynamodbDownloading and installing from sourceDownload the latest version ofscrapy-dynamodbfromhttp://pypi.python.org/pypi/scrapy-dynamodb/You can install it by doing the following,:$ tar xvfz scrapy-dynamodb-0.0.0.tar.gz
$ cd scrapy-dynamodb-0.0.0
# python setup.py install # as rootUsing the development versionYou can clone the git repository by doing the following:$ git clone git://github.com/acordiner/scrapy-dynamodb.gitSetupInpipelines.pyadd import statement:from scrapy_dynamodb import DynamoDbPipelineUsing scrapy-dynamodbTo usescrapy-dynamodbin your Scrapy project, add
the following line to yoursettings.py:ITEM_PIPELINES = {
    'scraper.pipelines.DynamoDbPipeline': 1,
    # replace scraper with your scrapy project name
}

AWS_ACCESS_KEY_ID = '<aws access key id>'
AWS_SECRET_ACCESS_KEY = '<aws secret access key>'
DYNAMODB_PIPELINE_REGION_NAME = 'us-east-1'
DYNAMODB_PIPELINE_TABLE_NAME = 'my_table'
DYNAMODB_ENDPOINT_URL = '<dynamodb endpoint url>' # optional  (e.g.: http://localhost:8000)That’s it!
Package: scrapyer-rabbitmq-scheduler
Scrapy分布式RabbitMQ调度器安装使用pip安装pip install scrapyer-rabbitmq-scheduler版本:2022/09/06 更新 1.1.2兼容调度器scrapy_rabbitmq_scheduler.scheduler.MqMixCache与中间件RabbitMQMiddleware说明: 由于MqMixCache采用双列队, response异常情况将不重新入MQ列队,RabbitMQMiddleware将自动失效2022/09/02 更新 1.1.0新增调度器scrapy_rabbitmq_scheduler.scheduler.MqMixCacheMQ列队组合内存列队,将原先SaaS调度器的广度优先调整为深度优先2021/11/25 更新 1.0.9解决pipelines管道文件链接超时问题使用第一步: 在你的项目中的settings.py中添加配置项# 指定项目的调度器
SCHEDULER = "scrapy_rabbitmq_scheduler.scheduler.MqMixCache"

# 指定rabbitmq的连接DSN
RABBITMQ_CONNECTION_PARAMETERS = 'amqp://guest:guest@localhost:5672/?heartbeat=0'

# 指定重试的http状态码(重新加回队列重试)
SCHEDULER_REQUEUE_ON_STATUS = [500,503]

# 指定下载器中间件, 确认任务是否成功
DOWNLOADER_MIDDLEWARES = {
    'scrapy_rabbitmq_scheduler.middleware.RabbitMQMiddleware': 999
}
# 指定item处理方式, item会加入到rabbitmq中
ITEM_PIPELINES = {
    'scrapy_rabbitmq_scheduler.pipelines.RabbitMQPipeline': 300,
}第二步: 修改Spider的继承类import scrapy
from scrapy_rabbitmq_scheduler.spiders import RabbitSpider

class CustomSpider(RabbitSpider):
    name = 'custom_spider'    
    queue_name = 'test_urls' # 指定任务队列的名称
    items_key = 'test_item' # 指定item队列名称

    def parse(self, response):
        item = ... # parse item
        yield item第三步: 将任务写入到RabbitMQ队列#!/usr/bin/env python
import pika
import settings

connection = pika.BlockingConnection(pika.URLParameters(settings.RABBITMQ_CONNECTION_PARAMETERS))
channel = connection.channel()

queue_key = 'test_urls'

# 读取文件中的链接并写入到队列中
with open('urls.txt') as f:
    for url in f:
        url = url.strip(' \n\r')
        channel.basic_publish(exchange='',
                        routing_key=queue_key,
                        body=url,
                        properties=pika.BasicProperties(
                            content_type='text/plain',
                            delivery_mode=2
                        ))

connection.close()urls.txthttp://www.baidu.com高级特色1. 支持消息优先级消息优先级的范围为0~255, 数字越大, 优先级越高yieldscrapy.Request(url,priority=优先级)则可以直接指定消息的优先级2. 队列持久化# settings.pyRABBITMQ_DURABLE=True# 是否持久化队列, True为持久化 False为非持久化, 默认True3. 消息确认# settings.pyRABBITMQ_CONFIRM_DELIVERY=True# 消息是否需要确认, True为需要, False为不需要, 默认是True4. 增加消息延时scrapy-rabbitmq-scheduler的消息延时是使用rabbitmq-delayed-message-exchange插件实现的, 所以在使用之前需要先安装以及开启这个插件rabbitmq-delayed-message-exchange:https://github.com/rabbitmq/rabbitmq-delayed-message-exchange在spider中开启延时队列# -*- coding: utf-8 -*-importscrapyfromscrapy_rabbitmq_scheduler.spidersimportRabbitSpiderfromexample.itemsimportArticleItemclassCcidcomSpider(RabbitSpider):....# 队列名称queue_name='ccidcom'# 是否是延迟队列is_delay_queue=True...is_delay_queue设置为True,则自动会开启延时使用延时yieldscrapy.Request('http://www.ccidcom.com/',callback=self.parse,meta={'_delay_time':10000})在meta中增加_delay_time, 指定延时毫秒数, 则自动生效TODO支持延时请求增加任务持久化配置
Package: scrapy-fake-useragent-fix
scrapy-fake-useragent-fix=====================Random User-Agent middleware based on`fake-useragent <https://pypi.python.org/pypi/fake-useragent>`__. Itpicks up ``User-Agent`` strings based on `usagestatistics <http://www.w3schools.com/browsers/browsers_stats.asp>`__from a `real world database <http://useragentstring.com/>`__.Installation-------------The simplest way is to install it via `pip`:pip install scrapy-fake-useragent-fixConfiguration-------------Turn off the built-in ``UserAgentMiddleware`` and add``RandomUserAgentMiddleware``.In Scrapy >=1.0:.. code:: pythonDOWNLOADER_MIDDLEWARES = {'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,}In Scrapy <1.0:.. code:: pythonDOWNLOADER_MIDDLEWARES = {'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None,'scrapy_fake_useragent.middleware.RandomUserAgentMiddleware': 400,}Configuring User-Agent type---------------------------There's a configuration parameter ``RANDOM_UA_TYPE`` defaulting to ``random`` which is passed verbatim to the fake-user-agent to random choose user agents. ``random``, ``chrome``, ``firefox``, ``safari``, ``internetexplorer`` are supported. If you want to choose from a specific device type, you can use a device prefix before browse type, such as ``desktop.chrome``, ``mobile.chrome``, only ``desktop``, ``mobile``, ``tablet`` are supported.Usage with `scrapy-proxies`---------------------------To use with middlewares of random proxy such as `scrapy-proxies <https://github.com/aivarsk/scrapy-proxies>`_, you need:1. set ``RANDOM_UA_PER_PROXY`` to True to allow switch per proxy2. set priority of ``RandomUserAgentMiddleware`` to be greater than ``scrapy-proxies``, so that proxy is set before handle UA.. |GitHub version| image:: https://badge.fury.io/gh/alecxe%2Fscrapy-fake-useragent.svg:target: http://badge.fury.io/gh/alecxe%2Fscrapy-fake-useragent.. |Requirements Status| image:: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements.svg?branch=master:target: https://requires.io/github/alecxe/scrapy-fake-useragent/requirements/?branch=masterConfiguring Fake-UserAgent fallback-----------------------------------There's a configuration parameter ``FAKEUSERAGENT_FALLBACK`` defaulting to``None``. You can set it to a string value, for example ``Mozilla`` or``Your favorite browser``, this configuration can completely disable anyannoying exception.
Package: scrapy-feedexporter-azure-blob
scrapy-feedexporter-azure is aScrapy Feed Exporter Storage Backendthat allows you to exportScrapy itemsto an Azure Blob Container.Using scrapy-feedexporter-azure-blobAdd a the following to your Scrapy settings:FEED_STORAGES = {"azure": "scrapy_feedexporter_azure_blob.AzureBlobFeedStorage"}

FEED_URI = "azure://accountname/container"

AZURE_ACCOUNT_NAME = "mycrawldata"  # this is the subdomain to https://*.blob.core.windows.net/

AZURE_ACCOUNT_KEY = "your_account_key"

AZURE_CONTAINER = 'sites'   # the name of the container (you should have already created it)

AZURE_FILENAME = 'bob.json'   # the name of the file as it will be in blob storageNote, you can define all of these settings when you run your crawler with the-scommand line switch:scrapy crawl <crawler_name>-sAZURE_ACCOUNT_KEY='<account_key>'-sAZURE_ACCOUNT_NAME='<account_name>'TODOHave the feed exporter properly parse the FEED_URI string:azure://account_name:api_key@container/filename.jsonDEVELOPER NOTESWhen deploying to pypi, just use twine
Package: scrapy-feedexporter-azure-blob-3
scrapy-feedexporter-azure is aScrapy Feed Exporter Storage Backendthat allows you to exportScrapy itemsto an Azure Blob Container.Using scrapy-feedexporter-azure-blobAdd a the following to your Scrapy settings:FEED_STORAGES = {"azure": "scrapy_feedexporter_azure_blob.AzureBlobFeedStorage"}

FEED_URI = "azure://accountname/container"

AZURE_ACCOUNT_NAME = "mycrawldata"  # this is the subdomain to https://*.blob.core.windows.net/

AZURE_ACCOUNT_KEY = "your_account_key"

AZURE_CONTAINER = 'sites'   # the name of the container (you should have already created it)

AZURE_FILENAME = 'bob.json'   # the name of the file as it will be in blob storage
Package: scrapy-feedexporter-sftp
scrapy-feedexporter-sftp is aScrapy Feed Exporter Storage Backendthat allows you to exportScrapy itemsto an SFTP server.Using scrapy-feedexporter-sftpAdd aFEED_STORAGESto your Scrapy settings:FEED_STORAGES = {"sftp": "scrapy_feedexporter_sftp.SFTPFeedStorage"}Define yourFEED_URIin Scrapy settings:FEED_URI = "sftp://user:password@some.server:/some/path/to/a/file"Testing scrapy-feedexporter-sftpInstall an ssh server, create a user and run:export FEEDTEST_SFTP_URI='sftp://user:password@localhost:/some/path/to/a/file'
export FEEDTEST_SFTP_PATH='/some/path/to/a/file'
tox
Package: scrapy-feed-storage-internetarchive
scrapy-feed-storage-internetarchiveThis is aStorage BackendforScrapyItem Feedsthat uploads feed files toarchive.orgwhen a scrape job ends.This was created to make it easy to archive data at the Internet Archive which you are authorised to distribute. e.g. to archive public data.UsageInstall the custom storage backendWe recommend the schemeinternetarchiveFEED_STORAGES = {
    "internetarchive": "feedstorage_internetarchive.storages.InternetArchiveStorage",
}Configure the Internet Archive metadata templateMetadata values can be specified using the settings keyFEED_STORAGE_INTERNETARCHIVE, e.g.FEED_STORAGE_INTERNETARCHIVE = {
    "metadata": {
        "mediatype": "data",
        "coverage": "South Africa",
        "title": "eTender Portal %(name)s %(time)s %(filetype)s",
    }
}Configure the storage for your feedsUse the Feed Exporter configuration with the URI scheme you used for installing the backend.The Internet Archive feed exporter should have the hostnamearchive.org, Internet Archive S3 API access key and secret in the username and password positions.Only one level of path is allowed. This will be used as the filename, and will be transformed into a unique identifier, meaning it should be unique on all of the Internet Archive. Including the scrape job timestamp in this path is useful for ensuring uniqueness.Extra parameters can be provided as query string parameters, which will then be templated into the metadata values.e.g.FEEDS = {
    "internetarchive://YourIAS3AccessKey:YourIAS3APISecretKey@archive.org/south-africa-%(name)s-%(time)s.csv?time=%(time)s&name=%(name)s&filetype=csv": {
        "format": "csv",
    },
    "internetarchive://YourIAS3AccessKey:YourIAS3APISecretKey@archive.org/south-africa-%(name)s-%(time)s.jsonlines?time=%(time)s&name=%(name)s&filetype=jsonlines": {
        "format": "jsonlines",
    },
}You probably don't want to put credentials into your project settings module, since it can then easily be discovered if added to source control. So try to set it in the environment where you will run your spider.ScrapinghubYou can set theFEEDSkey in scrapinghub by providing the value dictionary as JSON on a single line in your spider's Raw Settings. For the above example, you would add the following line in the scrapinghub Raw settings:FEEDS = {"internetarchive://YourIAS3AccessKey:YourIAS3APISecretKey@archive.org/south-africa-%(name)s-%(time)s.csv?time=%(time)s&name=%(name)s": {"format": "csv"}, "internetarchive://YourIAS3AccessKey:YourIAS3APISecretKey@archive.org/south-africa-%(name)s-%(time)s.jsonlines?time=%(time)s&name=%(name)s": { "format": "jsonlines" }}After saving, you should see it parsed into a key and value on the standard settings pane.
Package: scrapy-fingerprint
scrapy-fingerprint2023/11/07 scrapy-fingerprint==1.0fix : This allows the entire suite of downloader functionality to functionDescriptionScrapy - fingerprint is based on [curl_cffi] (https://github.com/yifeikong/curl_cffi), which is used in the packaging of scrapy TLS or JA3 fingerprints of simulated browser requestsgithub:https://github.com/tieyongjie/scrapy-fingerprintInstallationpipinstallscrapy_fingerprintUsageAfter creating the scrapy project, add the proxy by adding the following configuration in settings.py# proxy 链接配置PROXY_HOST='http-dynamic-S02.xiaoxiangdaili.com'PROXY_PORT=10030PROXY_USER='******'PROXY_PASS='******'And you also need to enable download handler inDOWNLOAD_HANDLERSin settings.pyDOWNLOAD_HANDLERS={'http':('scrapy_fingerprint.fingerprint_download_handler.''FingerprintDownloadHandler'),'https':('scrapy_fingerprint.fingerprint_download_handler.''FingerprintDownloadHandler'),}You can use scrapy.Request to make a request with a browser fingerprintimportscrapyyieldscrapy.Request(url=url,callback=self.parse)You can also add impersonate in FingerprintRequestimportscrapyyieldscrapy.Request(url,callback=self.parse,meta={"impersonate":"chrome107"})impersonatedefaultsto random browser fingerprints
Package: scrapy-folder-tree
scrapy-folder-treeThis is a scrapy pipeline that provides an easy way to store files and images using various folder structures.Supported folder structures:Given this scraped file:05b40af07cb3284506acbf395452e0e93bfc94c8.jpg, you can choose the following folder structures:Using file nameclass:scrapy-folder-tree.ImagesHashTreePipelinefull
├── 0
.   ├── 5
.   .   ├── b
.   .   .   ├── 05b40af07cb3284506acbf395452e0e93bfc94c8.jpgUsing crawling timeclass:scrapy-folder-tree.ImagesTimeTreePipelinefull
├── 0
.   ├── 11
.   .   ├── 48
.   .   .   ├── 05b40af07cb3284506acbf395452e0e93bfc94c8.jpgUsing crawling dateclass:scrapy-folder-tree.ImagesDateTreePipelinefull
├── 2022
.   ├── 1
.   .   ├── 24
.   .   .   ├── 05b40af07cb3284506acbf395452e0e93bfc94c8.jpgInstallationpipinstallscrapy_folder_treeUsageUse the following settings in your project:ITEM_PIPELINES={'scrapy_folder_tree.FilesHashTreePipeline':300}FOLDER_TREE_DEPTH=3
Package: scrapy-gooaye
Gooayeprojectshow gooayenews article weekly【news_notify】 2021/08/10 gooaye news weekly broadcast 
 https://gooaye.com/0805-gooaye-news-%e9%81%8b%e9%88%94%e8%bb%8a/
【news_notify】 2021/08/10 gooaye news weekly broadcast 
 https://gooaye.com/0803-gooaye-news-%e9%81%8b%e9%88%94%e8%bb%8a/Data Sourceshttps://gooaye.com/Getting StartedInstall Scrapy & Python 3.6.8WindowsRun$gitclonehttps://github.com/Bluesss30/Gooaye.git
web:pipenvrunscrapycrawlgooaye
$pipinstallpipenv
pipenvinstall
pipenvrunscrapycrawlgooaye./GooayeNews.batUpdate ScheduleWeekly @Tuesday 10:00amDuration: ~ 1 mins
Package: scrapy-googlechat
scrapy-googlechatSend crawl reports from Scrapy spiders to Google ChatUsageCreate a webhook in Google Chat:https://developers.google.com/hangouts/chat/how-tos/webhooksWithin your Scrapy project, install the package:pipinstallscrapy-googlechatRegister and configure the extension insettings.py:EXTENSIONS={...'scrapy_google_chat.GoogleChatBot':100,...}...GOOGLE_CHAT_WEBHOOK='YOUR_WEBHOOK'# Optional: change the image displayed in Google ChatGOOGLE_CHAT_IMAGE='https://img.icons8.com/ios/452/spider.png'You're good to go. Happy crawling!
Package: scrapy-hcf
This Scrapy spider middleware uses the HCF backend from Scrapinghub’s
Scrapy Cloud service to retrieve the new urls to crawl
and store back the links extracted.InstallationInstall scrapy-hcf usingpip:$ pip install scrapy-hcfConfigurationTo activate this middleware it needs to be added to theSPIDER_MIDDLEWARESdict, i.e:SPIDER_MIDDLEWARES = {
    'scrapy_hcf.HcfMiddleware': 543,
}And the following settings need to be defined:HS_AUTHScrapy Cloud API keyHS_PROJECTIDScrapy Cloud project ID (not needed if the spider is ran on dash)HS_FRONTIERFrontier name.HS_CONSUME_FROM_SLOTSlot from where the spider will read new URLs.Note thatHS_FRONTIERandHS_CONSUME_FROM_SLOTcan be overriden
from inside a spider using the spider attributeshs_frontierandhs_consume_from_slotrespectively.The following optional Scrapy settings can be defined:HS_ENDPOINTURL to the API endpoint, i.e:http://localhost:8003.
The default value is provided by the python-hubstorage package.HS_MAX_LINKSNumber of links to be read from the HCF, the default is 1000.HS_START_JOB_ENABLEDEnable whether to start a new job when the spider finishes.
The default isFalseHS_START_JOB_ON_REASONThis is a list of closing reasons,
if the spider ends with any of these reasons a new job will be started
for the same slot. The default is['finished']HS_NUMBER_OF_SLOTSThis is the number of slots that the middleware will use to store the new links.
The default is 8.UsageThe following keys can be defined in a Scrapy Request meta in order to control the behavior
of the HCF middleware:'use_hcf'If set toTruethe request will be stored in the HCF.'hcf_params'Dictionary of parameters to be stored in the HCF with the request fingerprint'qdata'data to be stored along with the fingerprint in the request queue'fdata'data to be stored along with the fingerprint in the fingerprint set'p'Priority - lower priority numbers are returned first. The default is 0The value of'qdata'parameter could be retrieved later usingresponse.meta['hcf_params']['qdata'].The spider can override the default slot assignation function by setting the
spiderslot_callbackmethod to a function with the following signature:def slot_callback(request):
    ...
    return slot
Package: scrapy-heroku
A package to assist with running scrapy on heroku. This is accomplished by providing
a custom application configuration atscrapy_heroku.app.applicationthat launches
the scrapyd web service using the PORT environment variable and a multi-process work
queue implemented on a Postgres database specified by the DATABASE_URL environment
variable.ConfigurationCreate a git repo that has a scrapy project at the root (scrapy.cfg should be at the
top level). Edit your scrapy.cfg to include the following:[scrapyd]
application = scrapy_heroku.app.application

[deploy]
url = http://<YOUR_HEROKU_APP_NAME>.herokuapp.com:80/
project = <YOUR_PROJECT_NAME>
username = <A_USER_NAME>
password = <A_PASSWORD>Add a requirements.txt file that includesscrapy-herokuin it. It is strongly
recommended that you version pin scrapy-heroku as well as the version of scrapy that
your project is developed against (pip freeze > requirements.txt). Finally create a
Procfile that consists of:web: scrapy serverMake sure you have a postgres database that has been promoted to DATABASE_URLProject page: <http://github.com/dmclain/scrapy-heroku>
Package: scrapy-html-storage
This is Scrapy downloader middleware that stores response HTMLs to disk.UsageTurn downloader on, e.g. specifying it insettings.py:DOWNLOADER_MIDDLEWARES = {
    'scrapy_html_storage.HtmlStorageMiddleware': 10,
}None of responses by default are saved to disk.
You must select for which requests the response HTMLs will be saved:def parse(self, response):
     """Processes start urls.

     Args:
         response (HtmlResponse): scrapy HTML response object.
     """
     yield scrapy.Request(
         'http://target.com',
         callback=self.parse_target,
         meta={
           'save_html': True,
         }
     )The file path where HTML will be stored is resolved with spider methodresponse_html_path. E.g.:class TargetSpider(scrapy.Spider):
    def response_html_path(self, request):
        """
        Args:
            request (scrapy.http.request.Request): request that produced the
                response.
        """
        return 'html/last_response.html'ConfigurationHTML storage downloader middleware supports such options:gzip_output(bool) - if True, HTML output will be stored in gzip format.
Default is False.save_html_on_status(list) - if not empty, sets list of response codes
whitelisted for html saving. If list is empty or not provided, all response
codes will be allowed for html saving.Sample:HTML_STORAGE = {
    'gzip_output': True,
    'save_html_on_status': [200, 202]
}