Package: rss-news-reader
Python RSS readerFinal task for EPAM Python Training 2021.09rss-news-readeris a command line utility that makes it easy to view RSS feeds in a readable format.Python 3.9 requiredTested on Windows and MacOSInstallation and usageYou can install it by running the following command:pip install rss-news-readerNow, you can run the utility in two ways:rss-news-reader {YOUR ARGUMENTS}
rss-reader {YOUR ARGUMENTS}ORClone github repository:git clone https://github.com/doppelmarker/HomeworkChange directory to/Homework/MarkKanaplianik/final_task.cd .../Homework/MarkKanaplianik/final_taskInstall necessary dependencies:pip install -r requirements.txtNow, provided, your current directory is/Homework/MarkKanaplianik/final_task, you can runrss_news_readeras a
package:python rss_news_reader
python -m rss_news_readeror, provided, your current directory is/Homework/MarkKanaplianik/final_task/rss_news_reader, you can directly run the
module:python rss_reader.pyAdditional dependenciesIn order to install additional dependency to make--check-urlswork, please, use the following command:pip install aiohttpFunctionalityTo see help message, please, use-h/--helpargument:rss-news-reader -h.usage: rss-news-reader [-h] [-v] [--verbose] [-c] [--json] [-l LIMIT] [-d DATE] [--to-html [FOLDER_PATH]] [--to-pdf [FOLDER_PATH]] [--to-epub [FOLDER_PATH]] [--check-urls]
                   [--clear-cache]
                   [source]

Pure Python command-line RSS reader.

positional arguments:
  source                   RSS URL

optional arguments:
  -h, --help               Show this help message and exit.
  -v, --version            Print version info.
  --verbose                Output verbose status messages.
  -c, --colorize           Print news in colorized mode.
  --json                   Print news as JSON.
  -l LIMIT, --limit LIMIT  Limit news amount to be processed.
  -d DATE, --date DATE     Get news published on a specific date from cache for further processing.
  --to-html [FOLDER_PATH]  Convert news to .html format and save it by the specified folder path (FOLDER_PATH can be omitted).
  --to-pdf [FOLDER_PATH]   Convert news to .pdf format and save it by the specified folder path (FOLDER_PATH can be omitted).
  --to-epub [FOLDER_PATH]  Convert news to .epub format and save it by the specified folder path (FOLDER_PATH can be omitted).
  --check-urls             Ensure URL represents an image (requires installation of additional dependency, use: pip install aiohttp).
  --clear-cache            Clear cache file on startup.Some notes:IMPORTANT:rss-news-readerutility name was chosen, becauserss-readerwas already taken
onhttps://pypi.org/. However, it is still possible to utilize the application usingrss-readerword:rss-reader {YOUR ARGUMENTS}when--clear-cacheis passed individually, cache gets cleared and application terminates;--check-urlsrequires internet connection; when passed, it produces async HTTP HEAD requests in order to define
whether such URLs
ashttps://s.yimg.com/uu/api/res/1.2/LDLfXhKlx.t2_f.QDSUtqw--~B/aD0yODEyO3c9NDIxODthcHBpZD15dGFjaHlvbg--/https://media.zenfs.com/en/ap.org/4c9d417a53588f0217535af5f47a2ab4are images; without passing this argument some URLs representing images may be ascribed tootherscategory of
resulting parsed feed; this might bring some inconvenience to the user if he was unable to naturally see the image
rather than its link inotherscategory;whenever several URLs with different parameters (?width=...&height=...) but leading to the exact one image are
encountered, then image with the best resolution is chosen in order to avoid data redundancy; it has vital importance
especially for file conversion, when the user doesn't want to see the same images, ones with better quality and others
less attractive.LoggingThere are 2 loggers:general-purposerss-news-readerapplication logger;configlogger.Messages with eitherWARNINGorERRORseverities arealwaysprinted torss_news_reader.logfile.configlogs are only printed to console.If--verboseargument isNOTpassed, then only messages with eitherWARNINGorERRORseverities
ofrss_news_readerare printed to console,configlogs are not printed to console.If--verboseargument is passed, then allrss_news_readerlogs are printed both to console and log file,
whileconfiglogs are printed to console.ConfigurationApplication creates several files:cache.json;rss_news_reader.log;converted to supported formats files:news.html/pdf/epubBy default, the application files are stored inside home directory in a freshly createdrss_news_readerfolder:- Windows: C:\Users\User\rss_news_reader
    or C:\Users\rss_news_reader
- Linux and MacOS: /home/rss_news_readerYou can change this by addingrss_news_reader.inifile either insiderss_news_readerpackagelocally, or inside
home directory.Ifrss_news_reader.inifiles are present both insidepackageand home directory, then one insidepackageoverrides one from home directory.The structure ofrss_news_reader.inifile is the following:[rss-reader]
DEFAULT_DIR_PATH =
LOG_DIR_PATH =
CACHE_DIR_PATH =
CONVERT_DIR_PATH =The directory path resolution order for storing files,from lowest to highest priority, can be found below.Forrss_news_reader.logfile:home directory -> DEFAULT_DIR_PATH -> LOG_DIR_PATHForcache.jsonfile:home directory -> DEFAULT_DIR_PATH -> CACHE_DIR_PATHFor converted to supported formats files likenews.html/pdf/epub:home directory -> DEFAULT_DIR_PATH -> CONVERT_DIR_PATH -> command line argumentsIfrss_news_reader.inifile was given an invalid path or the path was empty, then the directory path gets resolved in
the reversed order.Cache JSON structureCache represents a dictionary of URLs with according lists of dictionaries of items, preceded by a dictionary of feed
info.Example:{
    "https://news.yahoo.com/rss/": [
        {
            "title": "Yahoo News - Latest News & Headlines",
            "description": "The latest news and headlines from Yahoo! News. Get breaking news stories and in-depth coverage with videos and photos.",
            "link": "https://www.yahoo.com/news",
            "image": "http://l.yimg.com/rz/d/yahoo_news_en-US_s_f_p_168x21_news.png",
            "language": "en-US"
        },
        {
            "title": "California county closes In-N-Out over vaccine verification",
            "description": "",
            "link": "https://news.yahoo.com/california-county-closes-n-over-014048650.html",
            "author": "",
            "pubDate": "2021-10-27T01:40:48Z",
            "links": {
                "images": [],
                "audios": [],
                "others": [
                    "http://www.ap.org",
                    "https://s.yimg.com/uu/api/res/1.2/LDLfXhKlx.t2_f.QDSUtqw--~B/aD0yODEyO3c9NDIxODthcHBpZD15dGFjaHlvbg--/https://media.zenfs.com/en/ap.org/4c9d417a53588f0217535af5f47a2ab4"
                ]
            }
        },
        {
            "title": "Lap dances at Hazard homecoming: DonвЂ™t schools already have enough problems these days?",
            "description": "",
            "link": "https://news.yahoo.com/lap-dances-hazard-homecoming-don-170226073.html",
            "author": "",
            "pubDate": "2021-10-27T17:02:26Z",
            "links": {
                "images": [],
                "audios": [],
                "others": [
                    "https://www.kentucky.com",
                    "https://s.yimg.com/uu/api/res/1.2/ad5IuEyvQKF5s5.nr9jNdg--~B/aD0yMDI4O3c9MTE0MDthcHBpZD15dGFjaHlvbg--/https://media.zenfs.com/en/lexington_herald_leader_mcclatchy_articles_314/b7fb5b959d75cd96f58b434428d59ef1"
                ]
            }
        },
        ...
    ...Some notes:cache auto-update mechanisms are not implemented, thus it endlessly grows; in order to clear cache
file--clear-cacheargument is provided;--json-printed results are different from ones, stored in cache; user is usually not encouraged to explore and
modify cache file (though, he is not forbidden to do so), because it's not a part of the public interface, that's why
developers have a right to implement it in a handy manner for them, but not in a user-friendly manner,
whereas--jsonargument is a part of the user interface, that's why its output is user-friendly.--jsonoutput example:{
    "feeds": [
        {
            "title": "Yahoo News - Latest News & Headlines",
            "description": "The latest news and headlines from Yahoo! News. Get breaking news stories and in-depth coverage with videos and photos.",
            "link": "https://www.yahoo.com/news",
            "image": "http://l.yimg.com/rz/d/yahoo_news_en-US_s_f_p_168x21_news.png",
            "language": "en-US",
            "items": [
                {
                    "title": "California county closes In-N-Out over vaccine verification",
                    "description": "",
                    "link": "https://news.yahoo.com/california-county-closes-n-over-014048650.html",
                    "author": "",
                    "pubDate": "2021-10-27T01:40:48Z",
                    "links": {
                        "images": [],
                        "audios": [],
                        "others": [
                            "https://s.yimg.com/uu/api/res/1.2/LDLfXhKlx.t2_f.QDSUtqw--~B/aD0yODEyO3c9NDIxODthcHBpZD15dGFjaHlvbg--/https://media.zenfs.com/en/ap.org/4c9d417a53588f0217535af5f47a2ab4",
                            "http://www.ap.org"
                        ]
                    }
                },
                {
                    "title": "Lap dances at Hazard homecoming: DonвЂ™t schools already have enough problems these days?",
                    "description": "",
                    "link": "https://news.yahoo.com/lap-dances-hazard-homecoming-don-170226073.html",
                    "author": "",
                    "pubDate": "2021-10-27T17:02:26Z",
                    "links": {
                        "images": [],
                        "audios": [],
                        "others": [
                            "https://www.kentucky.com",
                            "https://s.yimg.com/uu/api/res/1.2/ad5IuEyvQKF5s5.nr9jNdg--~B/aD0yMDI4O3c9MTE0MDthcHBpZD15dGFjaHlvbg--/https://media.zenfs.com/en/lexington_herald_leader_mcclatchy_articles_314/b7fb5b959d75cd96f58b434428d59ef1"
                        ]
                    }
                },
                ...
        ...Why is there a list of feeds inside--jsonstructure, not just a single feed? Inside cache file there may be items
with the samepubDate, but they may belong to different feeds. So, when there are such items and a user
passes--date DATEargument which represents this exact date, then these several items are returned and attributed to
several newly createdFeedinstances. After that, theseFeedinstances are printed. Printing returned news could be
implemented without respect to the feeds they belong to, but in this case it would be hard to distinguish them.Parsing XMLXML is parsed by parser implemented from scratch, it exploits the idea of XMLtokenization, dom-tree is created from
tokens.Features:XML CDATAparsing support: whenever CDATA is encountered in XML, it gets recursively parsed and substituted by a
normal text in the final form.XML CDATA example link:https://rss.art19.com/apology-linedetectinginvalid XML: parser notifies user with a wide range of messages whenever invalid syntax or some mistake
was encountered in XML document.Invalid XML example:https://feedforall.com/sample.xmlIts fragment (notice tags order):<i><font color="#0000FF">Homework Assignments <br> School Cancellations <br> Calendar of Events <br> Sports Scores <br> Clubs/Organization Meetings <br> Lunches Menus </i></font>handlingcommented pieces: whenever commented piece like<!-- wp:html -->is encountered, it gets skipped.Tested RSS linksChannels like these are parsed correctly:http://rss.cnn.com/rss/edition.rsshttps://worldoftanks.ru/ru/rss/news/http://feeds.feedburner.com/welikedotacurl's User-Agentis used to access some RSS channels like this one:https://www.dotabuff.com/blog.rss<char inside text is parsed correctly, as well ascommented piecesare skipped properly:https://defenseofthepatience.libsyn.com/rssEmpty XML documentis handled correctly:https://www.joindota.com/feeds/newsBig channelsare parsed correctly:https://feeds.megaphone.fm/WWO3519750118https://feeds.simplecast.com/54nAGcIlCDATAis parsed correctly:https://rss.art19.com/apology-lineUser is notified ifinvalid XMLis encountered:https://feedforall.com/sample.xmlFeeds inRussianare handled completely correctly:https://rss.dw.com/xml/rss-ru-rushttps://people.onliner.by/feedhttps://brestcity.com/blog/feedhttps://rss.dw.com/xml/rss-ru-newshttps://lenta.ru/rss/top7https://www.liga.net/tech/battles/rss.xmlhttps://vse.sale/news/rssSome others:https://news.yahoo.com/rss/https://www.liquiddota.com/rss/news.xmlPlease, seeKnown problemssection below:https://www.theguardian.com/international/rsshttps://www.hyprgame.com/blog/category/dota2/feed/TestingModules tested:_caching.py_builder.py_parser.pyTest coverage is 51%.In order to run tests, please, install dependencies:pip install pytest pytest-covThen, provided,/Homework/MarkKanaplianik/final_taskis your current directory, please, use the following command:pytest --cov=rss_news_reader tests/Known problems:Some problems with PDF conversion exist:https://www.theguardian.com/international/rsserror saving to .pdf; this error happens because
feature-pdf-word-wrap: CJK;is being used inside.jinja2template; without using this feature long strings
wouldn't be wrapped on the next line and pdf would look clumsy;https://www.hyprgame.com/blog/category/dota2/feed/error saving to .pdf (for some reason FileNotFoundError is
raised
(No such file or directory), but both of them exist).Big feeds like this onehttps://feeds.megaphone.fm/WWO3519750118may get truncated when printing to console because of
console's chars amount native limitations;--colorizeworks console-specifically, which implies that in different terminals colorized text may look
differently.