Package: scrapy-feedstreaming
scrapy-feedstreamingScrapy live Streaming data.scrapy.extensions.feedexport.FeedExporterfork to export item during scraping. See
[https://medium.com/@alex_ber/scrapy-streaming-data-cdf97434dc15]See CHANGELOG.md for detail description.Getting HelpQuickStartpython3-mpipinstall-Uscrapy-feedstreamingInstalling from Githubpython3-mpipinstall-Uhttps://github.com/alex-ber/scrapy-feedstreaming/archive/master.zipOptionally installing tests requirements.python3-mpipinstall-Uhttps://github.com/alex-ber/scrapy-feedstreaming/archive/master.zip#egg=alex-ber-utils[tests]Or explicitly:wgethttps://github.com/alex-ber/scrapy-feedstreaming/archive/master.zip-Omaster.zip;unzipmaster.zip;rmmaster.zipAnd then installing from source (see below).Installing from sourcepython3-mpipinstall-rreq.txt# only installs "required" (relaxed)python3-mpipinstall.# only installs "required"python3-mpipinstall.[tests]# installs dependencies for testsAlternatively you install install from requirements file:python3-mpipinstall-rrequirements.txt# only installs "required"python3-mpipinstall-rrequirements-tests.txt# installs dependencies for testsFrom the directory with setup.pypython3setup.pytest#run all testsorpytestInstalling new versionSeehttps://docs.python.org/3.1/distutils/uploading.htmlpython3setup.pysdistuploadRequirementsscrapy-feedstreaming requires the following modules.Python 3.6+ChangelogScrapy live Streaming data.scrapy.extensions.feedexport.FeedExporterfork to export item during scraping. See
[https://medium.com/@alex_ber/scrapy-streaming-data-cdf97434dc15]All notable changes to this project will be documented in this file.#https://pypi.org/manage/project/scrapy-feedstreaming/releases/[Unrelased][0.0.1] - 12/07/2020AddedBuffering was added toitem_scraped().S3FeedStorage: you can specifyACLas query part of URI.S3FeedStorage: support ofregionis added.FEEDS:slot_key_param: New (not available in Scrapy itself) specify (global) function which takes item and spider as parameter
andslot_key. Given the item that is passed through the pipeline to what URI you want to send it.
Fall back to noop method â€” method that does nothing.FEEDS:buff_capacity: New (not available in Scrapy itself) â€” after what amount of item you want to export them.
The fall back value is 1._FeedSlotinstances are created from your settings. They are created per supplied URI.
Some extra (compare to Scraping) information is stored, namely:uri_templateâ€” it is available through public API get_slots() method, see below.spider_nameâ€” is used in public API get_slots() method to restrict returned slots for requested spider.buff_capacityâ€”bufferâ€™s capacity, if the number of item exceed this number the buffer is flushedbuffâ€” buffer where all items pending export are stored.FeedExportedthere is 1 extra public methodget_slots()â€” this method is used to get feed slotâ€™s information (see implementation note above). It is populated from the settings. For example, you can retrieve to either URI you will export the items.
Note:slot_keyis slot identifier as described above. If you have only 1 URI you can supply None for this value.You can retrieve feed slotâ€™s information only from your spider.It has optionalforce_create=Trueparameter.
If youâ€™re calling this method early in the Scrapy life-cycle feed slotâ€™s information may be not yet created.
In this case, the default behavior is to create this information and return it for you.
Ifforce_create=Falseis supplied you will receive an empty collection of feed slotâ€™s information.OnS3FeedStoragethere couple of public methods:botocore_sessionbotocore_clientbotocore_base_kwargsâ€” dict of minimal parameters forbotocore_client.put_object()method as supplied in settings.botocore_kwargsâ€” dict of all supplied parametersfor botocore_client.put_object()method as supplied in settings.
For example, if supplied, it will containACLparameter whilebotocore_base_kwargswill not contain it.ChangedYou can have multiple URI for exports.Logic of sending the item was moved from theclose_spider()toitem_scraped().back-port Fix missingstorage.store()calls inFeedExporter.close_spider()[https://github.com/scrapy/scrapy/pull/4626]back-port Fix duplicated feed logs [https://github.com/scrapy/scrapy/pull/4629]Removedremoved deprecated: fallback tobotolibrary ifbotocoreis not foundremoved deprecated: implicit retrieval of settings from the project â€” settings is passed explicitly now