Package: scrapy-rss
Tools to easy generateRSS feedthat contains each scraped item usingScrapy framework.Package works with Python 2.7, 3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9 and 3.10.If you use Python 3.3 then you have to use Scrapy<1.5.0.If you use Python 2.7 then you have to use Scrapy<2.0.Table of ContentsInstallationHow To UseConfigurationOptional Additional CustomizationUsageBasic usageRssItem derivation and namespacesScrapy Project ExamplesInstallationInstallscrapy_rssusing pippipinstallscrapy_rssor using pip for the specific interpreter, e.g.:pip3installscrapy_rssor using setuptools directly:cdpath/to/root/of/scrapy_rsspythonsetup.pyinstallor using setuptools for specific interpreter, e.g.:cdpath/to/root/of/scrapy_rsspython3setup.pyinstallHow To UseConfigurationAdd parameters to the Scrapy project settings (settings.pyfile)
or to thecustom_settingsattribute of the spider:Add item pipeline that export items to rss feed:ITEM_PIPELINES={# ...'scrapy_rss.pipelines.RssExportPipeline':900,# or another priority# ...}Add required feed parameters:FEED_FILEthe absolute or relative file path where the result RSS feed will be saved.
For example,feed.rssoroutput/feed.rss.FEED_TITLEthe name of the channel (feed),FEED_DESCRIPTIONthe phrase or sentence that describes the channel (feed),FEED_LINKthe URL to the HTML website corresponding to the channel (feed)FEED_FILE='path/to/feed.rss'FEED_TITLE='Some title of the channel'FEED_LINK='http://example.com/rss'FEED_DESCRIPTION='About channel'Feed (Channel) Elements Customization [optionally]If you want to change other channel parameters (such as language, copyright, managing_editor,
webmaster, pubdate, last_build_date, category, generator, docs, ttl)
then define your own exporter that’s inherited fromRssItemExporterclass, for example:fromscrapy_rss.exportersimportRssItemExporterclassMyRssItemExporter(RssItemExporter):def__init__(self,*args,**kwargs):kwargs['generator']=kwargs.get('generator','Special generator')kwargs['language']=kwargs.get('language','en-us')super(CustomRssItemExporter,self).__init__(*args,**kwargs)And addFEED_EXPORTERparameter to the Scrapy project settings
or to thecustom_settingsattribute of the spider:FEED_EXPORTER='myproject.exporters.MyRssItemExporter'UsageBasic usageDeclare your item directly as RssItem():importscrapy_rssitem1=scrapy_rss.RssItem()Or use predefined item classRssedItemwith RSS field named asrssthat’s instance ofRssItem:importscrapyimportscrapy_rssclassMyItem(scrapy_rss.RssedItem):field1=scrapy.Field()field2=scrapy.Field()# ...item2=MyItem()Set/get item fields. Case sensitive attributes ofRssItem()are appropriate to RSS elements.
Attributes of RSS elements are case sensitive too.
If the editor allows autocompletion then it suggests attributes for instances ofRssedItemandRssItem.
It’s allowed to setanysubset of RSS elements (e.g. title only). For example:fromdatetimeimportdatetimeitem1.title='RSS item title'# set value of <title> elementtitle=item1.title.title# get value of <title> elementitem1.description='description'item1.guid='item identifier'item1.guid.isPermaLink=True# set value of attribute isPermalink of <guid> element,# isPermaLink is False by defaultis_permalink=item1.guid.isPermaLink# get value of attribute isPermalink of <guid> elementguid=item1.guid.guid# get value of element <guid>item1.category='single category'category=item1.categoryitem1.category=['first category','second category']first_category=item1.category[0].category# get value of the element <category> with multiple valuesall_categories=[cat.categoryforcatinitem1.category]# direct attributes settingitem1.enclosure.url='http://example.com/file'item1.enclosure.length=0item1.enclosure.type='text/plain'# or dict based attributes settingitem1.enclosure={'url':'http://example.com/file','length':0,'type':'text/plain'}item1.guid={'guid':'item identifier','isPermaLink':True}item1.pubDate=datetime.now()# correctly works with Python' datetimesitem2.rss.title='Item title'item2.rss.guid='identifier'item2.rss.enclosure={'url':'http://example.com/file','length':0,'type':'text/plain'}All allowed elements are listed in thescrapy_rss/items.py.
All allowed attributes of each element with constraints and default values
are listed in thescrapy_rss/elements.py.
Also you can readRSS specificationfor more details.RssItemderivation and namespacesYou can extend RssItem to add new XML fields that can be namespaced or not.
You can specify namespaces in an attribute and/or an element constructors.
Namespace prefix can be specified in the attribute/element name
using double underscores as delimiter (prefix__name)
or in the attribute/element constructor usingns_prefixargument.
Namespace URI can be specified usingns_uriargument of the constructor.fromscrapy_rss.metaimportItemElementAttribute,ItemElementfromscrapy_rss.itemsimportRssItemclassElement0(ItemElement):# attributes without special namespaceattr0=ItemElementAttribute(is_content=True,required=True)attr1=ItemElementAttribute()classElement1(ItemElement):# attribute "prefix2:attr2" with namespace xmlns:prefix2="id2"attr2=ItemElementAttribute(ns_prefix="prefix2",ns_uri="id2")# attribute "prefix3:attr3" with namespace xmlns:prefix3="id3"prefix3__attr3=ItemElementAttribute(ns_uri="id3")# attribute "prefix4:attr4" with namespace xmlns:prefix4="id4"fake_prefix__attr4=ItemElementAttribute(ns_prefix="prefix4",ns_uri="id4")# attribute "attr5" with default namespace xmlns="id5"attr5=ItemElementAttribute(ns_uri="id5")classMyXMLItem(RssItem):# element <elem1> without namespaceelem1=Element0()# element <elem_prefix2:elem2> with namespace xmlns:elem_prefix2="id2e"elem2=Element0(ns_prefix="elem_prefix2",ns_uri="id2e")# element <elem_prefix3:elem3> with namespace xmlns:elem_prefix3="id3e"elem_prefix3__elem3=Element1(ns_uri="id3e")# yet another element <elem_prefix4:elem3> with namespace xmlns:elem_prefix4="id4e"# (does not conflict with previous one)fake_prefix__elem3=Element0(ns_prefix="elem_prefix4",ns_uri="id4e")# element <elem5> with default namespace xmlns="id5e"elem5=Element0(ns_uri="id5e")Access to elements and its attributes is the same as with simple items:item=MyXMLItem()item.title='Some title'item.elem1.attr0='Required content value'item.elem1='Another way to set content value'item.elem1.attr1='Some attribute value'item.elem_prefix3__elem3.prefix3__attr3='Yet another attribute value'item.elem_prefix3__elem3.fake_prefix__attr4=''# non-None value is interpreted as assigneditem.fake_prefix__elem3.attr1=42Several optional settings are allowed for namespaced items:FEED_NAMESPACESlist of tuples[(prefix, URI), ...]or dictionary{prefix: URI, ...}of namespaces
that must be defined in the root XML elementFEED_ITEM_CLASS or FEED_ITEM_CLSmain class of feed items (class objectMyXMLItemor path to class"path.to.MyXMLItem").Default value:RssItem.
It’s used in order to extract all possible namespaces
that will be declared in the root XML element.Feed items doNOThave to be instances of this class or its subclass.If these settings are not defined or only part of namespaces are defined
then other used namespaces will be declared either in the<item>element
or in its subelements when these namespaces are not unique.
Each<item>element and its sublements always contains
only namespace declarations of non-Noneattributes (including ones that are interpreted as element content).Scrapy Project ExamplesExamples directorycontains
several Scrapy projects with the scrapy_rss usage demonstration. It crawlsthis websitewhose source code ishere.Just go to the Scrapy project directory and run commandsscrapycrawlfirst_spiderscrapycrawlsecond_spiderThereafterfeed.rssandfeed2.rssfiles will be created in the same directory.