Saravanan666/pypi-org huggingface dataset:
Package: scrapy-scylla-proxies
Random proxy middleware forScrapyUsingScyllato fetch valid proxies.NOTE: I am not a 'real' programmer, help always appreciated! But it works! ... for now.Processes Scrapy requests using a random proxies to avoid IP ban and
improve crawling speed, this plugs in to the Scylla project which provides a local database of proxies.Install & run ScyllaThe Scylla project will need to be set-up separately!! The quickest way to do this is to use the docker container. The following command will download and run Scylla (provided you have docker installed of course).docker run -d -p 8899:8899 -p 8081:8081 --name scylla wildcat/scylla:latestInstall scrapy-scylla-proxiesThe quick way:pip install scrapy-scylla-proxiesOr checkout the source and runpython setup.py installWhat to put in Scrapy's 'settings.py'This is stuff you are going to need to integrate this middleware with Scrapy.SSP_ENABLED- ThisMUSTbe set to True.SSP_SCYLLA_URI- The location of the Scylla API (Default: 'http://localhost:8899').SSP_PROXY_TIMEOUT- How often the proxy list is refreshed (Default: 60s).SSP_HTTPS- Whether to only use HTTPS proxies, You will need this set to True if you are scraping an HTTPS site (Default: True).SSP_SPLASH_REQUEST_ENABLED- Whether this middleware will need to set the proxy for a 'scrapy.Request' or a 'SplashRequest' (Default: False)Example 'settings.py'This is a sample taken directly from a working scraper of mine, I used it to scrape approximately 15000 items from a website without any 'bans'.# Configure maximum concurrent requests performed by Scrapy (default: 16)
CONCURRENT_REQUESTS = 32

# Enable or disable downloader middlewares
# See https://doc.scrapy.org/en/latest/topics/downloader-middleware.html
DOWNLOADER_MIDDLEWARES = {
    # For retries
    'scrapy.downloadermiddlewares.retry.RetryMiddleware': 290,
    # For random scylla proxies
    'scrapy_scylla_proxies.random_proxy.RandomProxyMiddleware': 300,
    # For http proxy ip rotation
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
}

DOWNLOAD_TIMEOUT = 180
RETRY_TIMES = 10
RETRY_HTTP_CODES = [500, 502, 503, 504, 408, 400, 429, 403, 404]

# scrapy-scylla-proxies settings
# Enabled
SSP_ENABLED = True
# Location of the scylla server
SSP_SCYLLA_URI = 'http://localhost:8899'
# Proxy timeout in seconds
SSP_PROXY_TIMEOUT = 60
# Get only https proxies
SSP_HTTPS = TrueTipsI also find that rotating your user agent incombinationwith this middleware can be helpful in minimising failures due to being banned!DonateIf you like this middleware or it was helpful to you, you can always send me a small donation, even just a token amount. It will encourage me to keep developing this middleware and improving it! :::fire:::Donate here!
Package: scrapy-selenium-middleware
scrapy-selenium-middlewarerequirementsThis downloader middleware should be used inside an existingScrapyprojectInstall  Firefox andgekodriveron the machine running this middlewarepippip install scrapy-selenium-middlewareusage examplefor a full scrapy project demo please gohereThe middleware receives its settings fromscrapy project settingsin your scrapy project settings.py file add the following settingsDOWNLOADER_MIDDLEWARES={"scrapy_selenium_middleware.SeleniumDownloader":451}CONCURRENT_REQUESTS=1# multiple concurrent browsers are not supported yetSELENIUM_IS_HEADLESS=FalseSELENIUM_PROXY="http://user:password@my-proxy-server:port"# set to None to not use a proxySELENIUM_USER_AGENT="User-Agent: Mozilla/5.0 (<system-information>) <platform> (<platform-details>) <extensions>"SELENIUM_REQUEST_RECORD_SCOPE=["api*"]# a list of regular expression to record the incoming requests by matching the urlSELENIUM_FIREFOX_PROFILE_SETTINGS={}SELENIUM_PAGE_LOAD_TIMEOUT=120
Package: scrapy-selenium-mm
Scrapy with seleniumScrapy middleware to handle javascript pages using selenium.Installation$ pip install scrapy-seleniumYou should usepython>=3.6.
You will also need one of the Seleniumcompatible browsers.ConfigurationAdd the browser to use, the path to the driver executable, and the arguments to pass to the executable to the scrapy settings:fromshutilimportwhichSELENIUM_DRIVER_NAME='firefox'SELENIUM_DRIVER_EXECUTABLE_PATH=which('geckodriver')SELENIUM_DRIVER_ARGUMENTS=['-headless']# '--headless' if using chrome instead of firefoxOptionally, set the path to the browser executable:SELENIUM_BROWSER_EXECUTABLE_PATH=which('firefox')In order to use a remote Selenium driver, specifySELENIUM_COMMAND_EXECUTORinstead ofSELENIUM_DRIVER_EXECUTABLE_PATH:SELENIUM_COMMAND_EXECUTOR='http://localhost:4444/wd/hub'Add theSeleniumMiddlewareto the downloader middlewares:DOWNLOADER_MIDDLEWARES={'scrapy_selenium.SeleniumMiddleware':800}UsageUse thescrapy_selenium.SeleniumRequestinstead of the scrapy built-inRequestlike below:fromscrapy_seleniumimportSeleniumRequestyieldSeleniumRequest(url=url,callback=self.parse_result)The request will be handled by selenium, and the request will have an additionalmetakey, nameddrivercontaining the selenium driver with the request processed.defparse_result(self,response):print(response.request.meta['driver'].title)For more information about the available driver methods and attributes, refer to theselenium python documentationTheselectorresponse attribute work as usual (but contains the html processed by the selenium driver).defparse_result(self,response):print(response.selector.xpath('//title/@text'))Additional argumentsThescrapy_selenium.SeleniumRequestaccept 4 additional arguments:wait_time/wait_untilWhen used, selenium will perform anExplicit waitbefore returning the response to the spider.fromselenium.webdriver.common.byimportByfromselenium.webdriver.supportimportexpected_conditionsasECyieldSeleniumRequest(url=url,callback=self.parse_result,wait_time=10,wait_until=EC.element_to_be_clickable((By.ID,'someid')))screenshotWhen used, selenium will take a screenshot of the page and the binary data of the .png captured will be added to the responsemeta:yieldSeleniumRequest(url=url,callback=self.parse_result,screenshot=True)defparse_result(self,response):withopen('image.png','wb')asimage_file:image_file.write(response.meta['screenshot'])scriptWhen used, selenium will execute custom JavaScript code.yieldSeleniumRequest(url=url,callback=self.parse_result,script='window.scrollTo(0, document.body.scrollHeight);',)
Package: scrapy-selenium-python-pi
# Scrapy with seleniumScrapy middleware to handle javascript pages using selenium.## Installation```$ pip install scrapy_selenium_python_pi```You should use **python>=3.5**.You will also need one of the Selenium [compatible browsers](http://www.seleniumhq.org/about/platforms.jsp).## Configuration1. Add the browser to use, the path to the executable, and the arguments to pass to the executable to the scrapy settings:```pythonfrom shutil import whichSELENIUM_DRIVER_NAME='firefox'SELENIUM_DRIVER_EXECUTABLE_PATH=which('geckodriver')SELENIUM_DRIVER_ARGUMENTS=['-headless']  # '--headless' if using chrome instead of firefox```2. Add the `SeleniumMiddleware` to the downloader middlewares:```pythonDOWNLOADER_MIDDLEWARES = {'scrapy_selenium_python_pi.SeleniumMiddleware': 800}```## UsageUse the `scrapy_selenium_python_pi.SeleniumRequest` instead of the scrapy built-in `Request` like below:```pythonfrom scrapy_selenium_python_pi import SeleniumRequestyield SeleniumRequest(url, self.parse_result)```The request will be handled by selenium, and the response will have an additional `meta` key, named `driver` containing the selenium driver with the request processed.```pythondef parse_result(self, response):print(response.meta['driver'].title)```For more information about the available driver methods and attributes, refer to the [selenium python documentation](http://selenium-python.readthedocs.io/api.html#module-selenium.webdriver.remote.webdriver)The `selector` response attribute work as usual (but contains the html processed by the selenium driver).```pythondef parse_result(self, response):print(response.selector.xpath('//title/@text'))```### Additional argumentsThe `scrapy_selenium_python_pi.SeleniumRequest` accept 3 additional arguments:#### `wait_time` / `wait_until`When used, selenium will perform an [Explicit wait](http://selenium-python.readthedocs.io/waits.html#explicit-waits) before returning the response to the spider.```pythonfrom selenium.webdriver.common.by import Byfrom selenium.webdriver.support import expected_conditions as ECyield SeleniumRequest(url,self.parse_result,wait_time=10,wait_until=EC.element_to_be_clickable((By.ID, 'someid')))```#### `screenshot`When used, selenium will take a screenshot of the page and the binary data of the .png captured will be added to the response `meta`:```pythonyield SeleniumRequest(url,self.parse_result,screenshot=True)def parse_result(self, response):with open('image.png', 'wb') as image_file:image_file.write(response.meta['screenshot])```
Package: scrapy-sentry
scrapy-sentry=============Logs Scrapy exceptions into SentryA python library that glues [Sentry](http://www.getsentry.com) with [Scrapy](http://www.scrapy.org).Any spider errors will get pushed to Sentry. Please note that currently, exceptions anywhere else (e.g. the Scrapy pipeline)are not being reported to Sentry.Requisites:-----------* [Sentry server](http://www.getsentry.com/)Installation------------```pip install scrapy-sentry```Setup-----Add `SENTRY_DSN` and `scrapy_sentry.extensions.Errors` extension to your Scrapy Project `settings.py`.Example:```# sentry dsnSENTRY_DSN = 'http://public:secret@example.com/1'EXTENSIONS = {"scrapy_sentry.extensions.Errors":10,}```Supported versions------------------This package works with Python 2.7, 3.4 and 3.5. It has been tested with Scrapy up to version 1.2.1.
Package: scrapy-sentry-errors
🐞 scrapy-sentry-errorsA simple Scrapy extension that logs spider errors to your Sentry account, helping you monitor and fix issues with your Scrapy spiders efficiently. 🚀Note: While this extension captures errors from Scrapy spiders, it does not capture errors elsewhere in Scrapy's operation (e.g., the Scrapy pipeline).📋 RequirementsPython 3.8 or higherASentryaccount.TheDSNfor your Sentry project.🛠️ Installationpipinstallcity-scrapers-sentry🏗️ SetupAddSENTRY_DSNandscrapy_sentry_errors.extensions.Errorsextension to your Scrapy project'ssettings.pyfile:Example:# sentry dsn
SENTRY_DSN = 'http://public:secret@example.com/1'
EXTENSIONS = {
    "scrapy_sentry_errors.extensions.Errors": 10,
}Sentry spider errors will be logged to your Sentry account.💻 DevelopmentClone this repository.Run the dev setup script and follow the prompts. This will create a virtual environment and a.envfile with your Sentry DSN:bash ./scripts/dev_setup.shSimulate a failed spider run using the example Scrapy project in this repo:pipenv run example🧪 TestingRun the tests with:pipenv run test📦 PublishingBump the version number inpyproject.tomlCreate a new build with:pipenv run buildPublish the build to PyPI with:pipenv run publish🤝 ContributingWe welcome contributions of all kinds from anyone. Here are some ways you can contribute:Reporting Bugs: If you find a bug, please report it by opening an issue. Be sure to provide a detailed description of the bug and, if possible, steps to reproduce it.Suggesting Enhancements: Have an idea for a new feature or think something could be improved? Let us know by opening an issue.Code Contributions: Feel free to fork the repository and submit pull requests. Whether it's fixing a bug, adding a feature, or improving documentation, your help is welcome.🙏 AcknowledgementsThis project began as a fork ofscrapy-sentry, which was developed by Jordi Llonch. We are grateful for his work and those of other contributors.
Package: scrapy-sentry-sdk
scrapy-sentry-sdkA Scrapy extension for integration of Sentry SDK to Scrapy projects.This package provides a Scrapy extension for convenient initialization of Sentry SDK.Installationpipinstallscrapy_sentry_sdkUsageTo use the extension add the following to you projectsettings.py:# Send exceptions to Sentry# replace SENTRY_DSN by you own DSNSENTRY_DSN="XXXXXXXXXX"# Optionally, additional configuration options can be providedSENTRY_CLIENT_OPTIONS={"release":"you-project@version"# these correspond to the sentry_sdk.init kwargs}# Enable or disable extensions# See https://doc.scrapy.org/en/latest/topics/extensions.htmlEXTENSIONS={'scrapy_sentry_sdk.extensions.SentryLogging':1,# Load SentryLogging extension before others}ConfigurationCurrently, this extension uses two Scrapy settings keys:SENTRY_DSN: your project DSN (string, required)SENTRY_CLIENT_OPTIONS: additional SDK options (dict, optional)More details on configuring the SDK can be found inSentry documentation.
Package: scrapy-settings-log
Scrapy Settings LogAn extension that allows a user to display all or some of their scrapy spider settings at runtime.
It will add a logline with some or all settings for your spider in JSON compatible format.Installpip install scrapy-settings-logQuick-StartAdd the following to your settings:EXTENSIONS={'scrapy_settings_log.SpiderSettingsLogging':999,}SETTINGS_LOGGING_ENABLED=TrueWhen you run your spider you will see a log like below when spider is closing:[scrapy_settings_log] INFO: {"SETTINGS_LOGGING_ENABLED": true, ...}Additional OptionsSETTINGS_LOGGING_REGEX- Add a regular expression to only show some settings - for exampleSETTINGS_LOGGING_REGEX = "SPIDERMON"will show settings with SPIDERMON in their name.SETTINGS_LOGGING_INDENT- Add indentation to make log more human-readable.AdvancedSubclass and override theoutput_settingsmethod if you want the settings to be reported in another way.fromscrapy_settings_logimportSpiderSettingsLoggingclassCustomSettingsLogger(SpiderSettingsLogging):defoutput_settings(self,settings:dict,spider:scrapy.Spider):# custom code here
Package: scrapy-sink
Pipeline to post scrapy item to your urlInstallationInstall scrapy-sink using pip::$ pip install scrapy-sinkConfigurationAdd thesettings.pyof your Scrapy project like this:SINK_ADDR='http://127.0.0.1:8000/data_receiver.php'You can also only set it in your environment variableEnable the pipeline by adding it toITEM_PIPELINESin yoursettings.pyfile and changing HttpCompressionMiddleware
priority:ITEM_PIPELINES={'scrapy_sink.pipelines.SinkPipeline':9999,}The order should after your persist pipeline such as save to database and after your preprocess pipeline.Usageno need to change your codeGetting helpPlease use github issueContributingPRs are always welcomed.Changes0.1.0 (2020-12-26)Initial release
Package: scrapy-slackbot
scrapy-slackbotA Scrapy extension to send notification to the Slack channel.Installation$pipinstallscrapy-slackbotSettingsInsettings.py:EXTENSIONS={'scrapyslackbot.extensions.SlackBot':500,}SLACK_ENABLED=TrueSLACK_BOT_TOKEN=''SLACK_CHANNEL=''SLACK_ENABLEDA boolean which specifies whether the bot is enabled.SLACK_BOT_TOKENBot User OAuth Access Tokengenerated bySlack.Follow the instructions ofSlack API, visitFeatures > OAuth & Permissionsand generate the token.SLACK_CHANNELThe ID of the channel in your workspace.Development$pdminstallDemoLicenseMIT
Package: scrapy-spiderdocs
Usage examplepipinstallscrapy-spiderdocsscrapyspiderdocs<module.name>Example projectSeedocumentedproject for example.# -*- coding: utf-8 -*-importscrapyclassExampleSpider(scrapy.Spider):"""Some text.
    Hi!

    ; Note

    Some note.

    ; Output

    {
        "1": 1
    }
    """name='example'allowed_domains=('example.com',)start_urls=('http://example.com/',)defparse(self,response):yield{'body_length':len(response.body)}classExampleSpider2(scrapy.Spider):"""Some text.
    Hi!

    ; Info

    Some info.
    """name='example2'allowed_domains=('example.com',)start_urls=('http://example.com/',)defparse(self,response):yield{'success':True}Settings:SPIDERDOCS_SECTION_PROCESSORS={'output':lambdaname,content:'###{name}\n\n```json\n{content}\n```'.format(name=name,content=content),'info':lambdaname,content:'{content}'.format(content=content)}Execute the command:scrapyspiderdocsdocumented.spidersOutput:# documented.spiders spiders

## example2 [documented.spiders.example.ExampleSpider2]

Some info.

## example [documented.spiders.example.ExampleSpider]

### Note

Some note.

### Output

```json
{
    "1": 1
}
```Output optionsstdoutscrapyspiderdocs<module.name>>somefile.md-o(–output) optionscrapyspiderdocs<module.name>-osomefile.mdSettingsSPIDERDOCS_LOCATIONS={'module.name':"somefile.md"}The setting used if no module specified.scrapyspiderdocsDocstring syntaxUse;to create sections. For example:; Section 1

Some text ...

; Section 2

Some text ...Use; endto close a section:This text will not be added to the documentation.

; Section 1

Some text ...

; end

And this text also will be skipped.Section processorsAn example:SPIDERDOCS_SECTION_PROCESSORS={'output':lambdaname,content:'###{name}\n\n```json\n{content}\n```'.format(name=name,content=content)};Output{"attr":"value"}will be translated into:### Output

```json
{
    "attr": "value"
}
```Scrapy settingsSPIDERDOCS_LOCATIONS: {<module>: <destination>}, default:{}.SPIDERDOCS_SECTION_PROCESSORS: {<section_name>: <function(name, content)->str>}, default:{}.See usage examples above.Developmentgitclonegit@github.com:nanvel/scrapy-spiderdocs.gitcdscrapy-spiderdocsvirtualenv.env--no-site-packages-p/usr/local/bin/python3source.env/bin/activatepipinstallscrapyscrapycrawlexamplescrapyspiderdocsdocumented.spiderspython-munittestdocumented.testsTODOunittests (is there is no docstring, …)
Package: scrapy-spiderstats-extension
Scrapy SpiderStats Extension将 Spider Stats 存储到 MongoDB 的扩展，可以用于爬虫监控和统计。安装pipinstallscrapy-spiderstats-extension使用在settings.py配置文件中开启SpiderStatsEXTENSIONS={"scrapyspiderstats.SpiderStats":0}STATS_MONGODB_URI="mongodb://localhost:27017"STATS_MONGODB_DB="scrapy"STATS_MONGODB_COL="spiderstats"存储结果启动状态{"_id":"5fb23d9cbaf515d71d3a9c6c","log_count/INFO":9,"start_time":"2020-11-16T08:51:40.705Z","stats_id":"2b55df7b46a548269ca603bb7ad889b2","spider_name":"test","pages":0,"pagerate":0,"items":0,"itemrate":0,"record_time":"2020-11-16T08:51:40.706Z"}记录状态{"_id":"5fb23dd8baf515d71d3a9c6d","log_count/INFO":12,"start_time":"2020-11-16T08:51:40.705Z","stats_id":"2b55df7b46a548269ca603bb7ad889b2","spider_name":"test","pages":510,"pagerate":510,"items":0,"itemrate":0,"record_time":"2020-11-16T08:52:40.713Z","log_count/DEBUG":1034,"scheduler/enqueued/redis":521,"scheduler/dequeued/redis":520,"downloader/request_count":520,"downloader/request_method_count/GET":520,"downloader/request_bytes":239235,"downloader/response_count":510,"downloader/response_status_count/200":510,"downloader/response_bytes":110675,"response_received_count":510,"downloader/exception_count":3,"downloader/exception_type_count/twisted-internet-error-TimeoutError":3,"retry/count":3,"retry/reason_count/twisted-internet-error-TimeoutError":3}完成状态{"_id":"5fb23e2ebaf515d71d3a9c6f","log_count/INFO":16,"start_time":"2020-11-16T08:51:40.705Z","stats_id":"2b55df7b46a548269ca603bb7ad889b2","spider_name":"test","pages":1000,"pagerate":6,"items":0,"itemrate":0,"record_time":"2020-11-16T08:54:06.125Z","log_count/DEBUG":2015,"scheduler/enqueued/redis":1007,"scheduler/dequeued/redis":1007,"downloader/request_count":1007,"downloader/request_method_count/GET":1007,"downloader/request_bytes":463763,"downloader/response_count":1000,"downloader/response_status_count/200":1000,"downloader/response_bytes":216997,"response_received_count":1000,"downloader/exception_count":7,"downloader/exception_type_count/twisted.internet.error.TimeoutError":7,"retry/count":7,"retry/reason_count/twisted.internet.error.TimeoutError":7,"elapsed_time_seconds":145.420645,"finish_time":"2020-11-16T08:54:06.125Z","finish_reason":"finished"}
Package: scrapy-spider-utils
打包确保我们已安装最新setuptools 和 wheel和twine ，下面是安装/更新命令python3-mpipinstall--user--upgradesetuptoolswheeltwine打包的我们的库/项目python3-m此时在当前目录我们会看到以下：dist/
  example_pkg_your_username-0.0.1-py3-none-any.whl
  example_pkg_your_username-0.0.1.tar.gz使用 twine 将打包好的库/项目上传到PYPI（需用到PYPI帐号密码）（此时只是上传到PYPI测试服，还不能 pip install 这个库/项目）python3-mtwineupload--repository-urlhttps://test.pypi.org/legacy/dist/*我们会看到如下界面：Uploadingdistributionstohttps://test.pypi.org/legacy/
Enteryourusername:[yourusername]Enteryourpassword:
Uploadingexample_pkg_your_username-0.0.1-py3-none-any.whl100%|█████████████████████|4.65k/4.65k[00:01<00:00,2.88kB/s]Uploadingexample_pkg_your_username-0.0.1.tar.gz100%|█████████████████████|4.25k/4.25k[00:01<00:00,3.05kB/s]上传成功之后，我们可以去PYPI的测试服查看是否上传成功，能上传成功的话就说明肯定也能成功上传到PYPI正式服（附：PYPI测试服地址）PYPI测试服的管理员会不定期删除上边的库，正式投入使用还是得上传到正式服。由于我先前有上传库到测试服，我们可以尝试搜索看看若是想测试下上传到测试服的库能否使用，可以使用如下命令python3-mpipinstall--index-urlhttps://test.pypi.org/simple/--no-depsexample-pkg-your-username大致会出现以下：Collectingexample-pkg-your-usernameDownloadinghttps://test-files.pythonhosted.org/packages/.../example-pkg-your-username-0.0.1-py3-none-any.whl
Installingcollectedpackages:example-pkg-your-username
Successfullyinstalledexample-pkg-your-username-0.0.1若是从测试服安装的我们的库能正常使用，那么我们就可以开始行动把它上传到PYPI正式服供大家使用了。（之所以特地提出这一步，是因为第一次上传库时，我们总会因为目录结构不会、未打包成库可正常使用打包了却不能用等等原因导致上传的是个“失败的库”，这样能避免别人会安装到我们的失败库）简单测试是否能正常使用直接如下即可，但具体里边的功能能否正常用我们还需调用一下，此处不做介绍>>>importexample_pkg>>>example_pkg.name'example_pkg'【重头戏】将库上传到 PYPI正式服twineuploaddist/*上传成功后该库即可直接pip安装如果对目录结构或者其他有什么不清楚的可以参考我这个库（结构较简单适合初学者），或者我们平时使用的库（譬如本人平时经常使用 scrapy 也可以去 scrapy 主页参考大佬的写法）
Package: scrapysplashwrapper
ScrapySplashWrapperA wrapper that uses scrappy and splash to crawl a website.UsageWarning: it requires a splash instance (docker is recommendended).usage: scraper [-h] [-s SPLASH] -u URL [-d DEPTH] [-o OUTPUT] [-ua USERAGENT]
               [--debug]

Crawl a URL.

optional arguments:
  -h, --help            show this help message and exit
  -s SPLASH, --splash SPLASH
                        Splash URL to use for crawling.
  -u URL, --url URL     URL to crawl
  -d DEPTH, --depth DEPTH
                        Depth of the crawl.
  -o OUTPUT, --output OUTPUT
                        Output directory
  -ua USERAGENT, --useragent USERAGENT
                        User-Agent to use for crawling
  --debug               Enable debug mode on scrapy/splash
Package: scrapy-splitvariants
SplitVariantsMiddlewareis a Scrapy spider middleware used to split
single items into multiple items when they have a “variants” key with multiple values.Example usageLet’s assume your spider outputs an item with different size options
(from an ecommerce website for example):item = {"id": 12,
        "name": "Big chair",
        "variants": [{"size": "XL", "price": 200, "currency": "USD"},
                     {"size": "L", "price": 100, "currency": "USD"}]}When you enableSplitVariantsMiddleware, this single item will become
2 items with the different variants values into a different item:{"id": 12, "name": "Big chair", "size": "XL", "price": 200, "currency": "USD"}
{"id": 12, "name": "Big chair", "size": "L", "price": 100, "currency": "USD"}InstallationInstall scrapy-splitvariants usingpip:$ pip install scrapy-splitvariantsConfigurationAddSplitVariantsMiddlewareby including it inSPIDER_MIDDLEWARESin yoursettings.pyfile:SPIDER_MIDDLEWARES = {
    'scrapy_splitvariants.SplitVariantsMiddleware': 100,
}Here, priority100is just an example.
Set its value depending on other middlewares you may have enabled already.Enable the middleware usingSPLITVARIANTS_ENABLEDset toTruein yoursetting.py.