Package: feedsearch-crawler
Feedsearch CrawlerFeedsearch Crawler is a Python library for searching websites forRSS,Atom, andJSONfeeds.It is a continuation of my work onFeedsearch, which is itself a continuation of the work done byDan Foreman-MackeyonFeedfinder2, which in turn is based onfeedfinder- originally written byMark Pilgrimand subsequently maintained byAaron Swartzuntil his untimely death.Feedsearch Crawler differs with all of the above in that it is now built as an asynchronousWeb crawlerforPython 3.7and above, usingasyncioandaiohttp, to allow much more rapid scanning of possible feed urls.An implementation using this library to provide a public Feed Search API is available athttps://feedsearch.devPull requests and suggestions are welcome.InstallationThe library is available onPyPI:pip install feedsearch-crawlerThe library requires Python 3.7+.UsageFeedsearch Crawler is called with the single functionsearch:>>>fromfeedsearch_crawlerimportsearch>>>feeds=search('xkcd.com')>>>feeds[FeedInfo('https://xkcd.com/rss.xml'),FeedInfo('https://xkcd.com/atom.xml')]>>>feeds[0].urlURL('https://xkcd.com/rss.xml')>>>str(feeds[0].url)'https://xkcd.com/rss.xml'>>>feeds[0].serialize(){'url':'https://xkcd.com/rss.xml','title':'xkcd.com','version':'rss20','score':24,'hubs':[],'description':'xkcd.com: A webcomic of romance and math humor.','is_push':False,'self_url':'','favicon':'https://xkcd.com/s/919f27.ico','content_type':'text/xml; charset=UTF-8','bozo':0,'site_url':'https://xkcd.com/','site_name':'xkcd: Chernobyl','favicon_data_uri':'','content_length':2847}If you are already running in anasyncio event loop, then you can import and awaitsearch_asyncinstead. Thesearchfunction is only a wrapper that runssearch_asyncin a new asyncio event loop.fromfeedsearch_crawlerimportsearch_asyncfeeds=awaitsearch_async('xkcd.com')A search will always return a list ofFeedInfoobjects, each of which will always have aurlproperty, which is aURLobject that can be decoded to a string withstr(url).
The returnedFeedInfoare sorted by thescorevalue from highest to lowest, with a higher score theoretically indicating a more relevant feed compared to the original URL provided. AFeedInfocan also be serialized to a JSON compatible dictionary by calling it's.serialize()method.The crawl logs can be accessed with:importlogginglogger=logging.getLogger("feedsearch_crawler")Feedsearch Crawler also provides a handy function to output the returned feeds as anOPMLsubscription list, encoded as a UTF-8 bytestring.fromfeedsearch_crawlerimportoutput_opmloutput_opml(feeds).decode()Search Argumentssearchandsearch_asynctake the following arguments:search(url:Union[URL,str,List[Union[URL,str]]],crawl_hosts:bool=True,try_urls:Union[List[str],bool]=False,concurrency:int=10,total_timeout:Union[float,aiohttp.ClientTimeout]=10,request_timeout:Union[float,aiohttp.ClientTimeout]=3,user_agent:str="Feedsearch Bot",max_content_length:int=1024*1024*10,max_depth:int=10,headers:dict={"X-Custom-Header":"Custom Header"},favicon_data_uri:bool=True,delay:float=0)url:Union[str, List[str]]: The initial URL or list of URLs at which to search for feeds. You may also provideURLobjects.crawl_hosts:bool: (default True): An optional argument to add the site host origin URL to the list of initial crawl URLs. (e.g. add "example.com" if crawling "example.com/path/rss.xml"). IfFalse, site metadata and favicon data may not be found.try_urls:Union[List[str], bool]: (default False): An optional list of URL paths to query for feeds. Takes the origins of theurlparameter and appends the provided paths. If no list is provided, buttry_urlsisTrue, then a list of common feed locations will be used.concurrency:int: (default 10): An optional argument to specify the maximum number of concurrent HTTP requests.total_timeout:float: (default 30.0): An optional argument to specify the time this function may run before timing out.request_timeout:float: (default 3.0): An optional argument that controls how long before each individual HTTP request times out.user_agent:str: An optional argument to override the default User-Agent header.max_content_length:int: (default 10Mb): An optional argument to specify the maximum size in bytes of each HTTP Response.max_depth:int: (default 10): An optional argument to limit the maximum depth of requests while following urls.headers:dict: An optional dictionary of headers to pass to each HTTP request.favicon_data_uri:bool: (default True): Optionally control whether to fetch found favicons and return them as a Data Uri.delay:float: (default 0.0): An optional argument to delay each HTTP request by the specified time in seconds. Used in conjunction with the concurrency setting to avoid overloading sites.FeedInfo ValuesIn addition to theurl, FeedInfo objects may have the following values:bozo:int: Set to 1 when feed data is not well formed or may not be a feed. Defaults 0.content_length:int: Current length of the feed in bytes.content_type:str:Content-Typevalue of the returned feed.description:str: Feed description.favicon:URL:URLof feed or siteFavicon.favicon_data_uri:str:Data Uriof Favicon.hubs:List[str]: List ofWebsubhubs of feed if available.is_podcast:bool: True if the feed contains validpodcastelements and enclosures.is_push:bool: True if feed contains valid Websub data.item_count:int: Number of items currently in the feed.last_updated:datetime: Date of the latest published entry.score:int: Computed relevance of feed url value to provided URL. May be safely ignored.self_url:URL:ref="self"value returned from feed links. In some cases may be different from feed url.site_name:str: Name of feed's website.site_url:URL:URLof feed's website.title:str: Feed Title.url:URL:URLlocation of feed.velocity:float: Mean number of items per day in the feed at the current time.version:str: Feed versionXML values,
orJSON feed.