Package: scrapy-podcast-rss
scrapy-podcast-rssThis package provides a Scrapy pipeline and items to generate a podcast RSS feed
from scraped information. It also allows to save the content locally or in an S3
bucket. You can then point your podcast player to the URL of the file and
listen to its content.InstallationInstall scrapy-podcast-rss usingpip:$pipinstallscrapy-podcast-rssConfigurationYou need to defineOUTPUT_URIin yoursettings.pyfile, this will
determine where your feed will be stored. For example:OUTPUT_URI='./my-podcast.xml'# Local file.OUTPUT_URI='s3://my-bucket/my-podcast.xml'# S3 bucket (read note on S3 storage).AddPodcastPipelineinITEM_PIPELINESin yoursettings.pyfile:ITEM_PIPELINES={'scrapy_podcast_rss.pipelines.PodcastPipeline':300,}Usagescrapy-podcast-rss defines two special items:PodcastDataItem: Stores information about the podcast.PodcastEpisodeItem: Stores information about each episode of the podcast.You must yield onePodcastDataItemand onePodcastEpisodeItemfor each
episode you want to export, before your spider closes.Here is the information you can currently store in each item (you need to use
the same names):PodcastDataItem:title: Title of the podcast.description: Description of the podcast.url: URL referencing the podcast.image_url: Main image of the podcast.PodcastDataItem:title: Title of the episode.description: Description of the episode.publication_date: Date of publication (datetimeobject with timezone).audio_url: URL of the audio.guid: Unique identifier of the episode.ExampleYou can find a minimal example of a spider using this package here:scrapy-podcast-rss-example.You can also find an example of the package being used in an AWS Lambda function here:scrapy-podcast-rss-serverless.Spider exampleimportdatetimeimportscrapyimportpytzfromscrapy_podcast_rssimportPodcastEpisodeItem,PodcastDataItemclassSimpleSpider(scrapy.Spider):name="simple_spider"start_urls=['http://example.com/']custom_settings={'OUTPUT_URI':'./my-podcast.xml','ITEM_PIPELINES':{'scrapy_podcast_rss.pipelines.PodcastPipeline':300,}}defparse(self,response):podcast_data_item=PodcastDataItem()podcast_data_item['title']="Podcast title"podcast_data_item['description']="Description of the podcast."podcast_data_item['url']="Podcast's URL"podcast_data_item['image_url']="https://live.staticflickr.com/4211/35400224382_9edcb984e5_c.jpg"# Sample imageyieldpodcast_data_itemepisode_item=PodcastEpisodeItem()episode_item['title']="Episode title"episode_item['description']="Episode description"pub_date_tz=datetime.datetime.strptime("01/01/2020","%m/%d/%Y").replace(tzinfo=pytz.UTC)episode_item['publication_date']=pub_date_tz# Publication date NEEDS to have a TIME ZONE.episode_item['guid']="Episode guid"# Simulated identifier.episode_item['audio_url']="https://ia801803.us.archive.org/13/items/MOZARTSerenadeEineKleineNachtmusikK."\"525-NEWTRANSFER01.I.Allegro/01.I.Allegro.mp3 "# Sample audio url.yieldepisode_itemNote on using S3 as storageTo use S3 storage locations, you can install scrapy-podcast-rss by doing:$pipinstallscrapy-podcast-rss[s3_storage]This will simply includeboto3in the dependencies.Once installed, you will need to have your credentials configured
(boto3 quickstart guide).