Package: news_fetcher
news_fetcherScript to fetch news URLs from news websites to database using API.InstallationInstall Python 3.8 or higher, installpoetry, runpoetry install --no-dev.Then you can just runpoetry run COMMANDto run specific commands under python virtual environment created by poetry.Or you can enter poetry shell (by runningpoetry shell) and then type script commands.You can also usepip.Installation exampleAssuming Python 3.8 or higher and poetry are installed.Initialise and update virtual environment (assuming you are in the folder with this README file):poetryinstall--no-devRun script:poetryrunpythonnews_fetcher/news_fetcher.py--helpWindows installation exampleAssuming Python 3.8 or higher is installed.Install poetry (in Windows PowerShell):(Invoke-WebRequest-Urihttps://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py-UseBasicParsing).Content|pythonYou may need to restart PowerShell or reboot your computer.To install or update libraries, run batch fileupdate.bat.Run script:poetryrunpythonnews_fetcher/news_fetcher.py--helpFilesrun_all.shis the Shell script for running all steps. It requires that environment variables are set in.envfile:MEDIAWIKI_CREDENTIALS,DATABASE_URL,WIKI_TOOL_DIRECTORY,DATA_FILE,SOURCE_PATH,SOURCE_NAME,TARGET_API_URL,WIKI_PREFIX,BOT_NAME,REQUESTS_INTERVAL.news_fetcher/news_fetcher.pyis the script entry point.news_fetcher/db.pyis the DB initialization module.news_fetcher/models.pyis the module with DB models.news_fetcher/module.pyis the module with base class for "source modules" which are used to grab news from different sources.Prostoprosport source moduleThis modules fetches news usingprostoprosport.ruAPI.news_fetcher/prostoprosport.pyis the source module.data/categories_from_js.jsonis a category URL data grabbed from JS.data/categories_bonus.jsonis an additional category URL data grabbed from RSS.RSS source moduleThis modules fetches news using RSS.news_fetcher/rss.pyis the source module.DB modelsSourceSource website.slug_name— string website ID (primary key), for example:birmingham-post.TagTag for news articles.tag_id— numerical ID (primary key).title— tag text, for example:Sport(must be unique).ArticleNews article from source website.article_id— numerical ID (primary key).source— source website (foreign key).slug_name— string identifier (must be unique per source website), for example:sir-stanley-matthews-1915-2000-a-potteries-hero.title— human-readable article title, for example:Sir Stanley Matthews 1915-2000: A Potteries hero; Stanley stayed loyal to his beloved.date— publication date, for example:2020-02-24T00:00:00.source_url— full article URL, for example:https://www.thefreelibrary.com/Sir+Stanley+Matthews+1915-2000%3A+A+Potteries+hero%3B+Stanley+stayed...-a060517953.source_url_ok—trueif URL can be retrieved,falseif it can not,nullif it was not checked yet.author_name— human-readable author name, may benull.wikitext_paragraphs— article content converted into wiki-text stored as JSON list of paragraphs, may benullif not fetched yet.misc_data— miscellaneous data stored as JSON, specific format and structure is module-dependent.tags— article tags (many-to-many relationwithTagmodel through technicalArticleTagmodel with table namedarticle_m2m_tag).UsageGetting help--helpShow help message and exit. If this option is used with command, then help message for that specific command will be printed.Common options--source-module TEXT(required) — source module name, can beprostoprosportorrssProstoprosport module options--data-file FILENAME— file with categories data (can be built usingprocess-categoriescommand)--source-path TEXT— API method name, can benewsormain_newsRSS module options--data-file FILENAME— JSON file with configuration, should contain folllowing keys:css_selector— CSS selector for article paragraphs on web pagesource_title— source titlesource_template_name— template name for generated wiki-pages (optional, source title is used by default)removed_last_lines— count of paragraphs at the end of article that should be skipped (optional, 0 by default)disable_bold_font—trueto avoid bold font in generated page (optional,falseby default)extra_first_lines— array of strings to add at the beginning of generated page (optional, empty by default)--source-name(required) — source slug name (identifier) for DB--source-path TEXT(required) — RSS feed URLCommandfetch-newsFetch news for page range and write data to DB. Pages are numbered from most recent (1) to least recent. Note that page numbers are now used inProstoprosportsource module only.Options--first-page INTEGER— number of first page to load, should not be less than 1--last-page INTEGER— number of last page to load, should not be less than 1. If it is less than first page number, no data will be fetchedExample 1Fetch most recent page (1):pythonnews_fetcher/prostoprosport_news_fetcher.pyfetch-newsExample 2Fetch pages 5 most recent pages (5 to 1):pythonnews_fetcher/prostoprosport_news_fetcher.pyfetch-news--last-page5Example 3Fetch pages 11 to 20:pythonnews_fetcher/prostoprosport_news_fetcher.pyfetch-news--first-page11--last-page20Notes(OBSOLETE) Prostoprosport.ru API did not provide URLs, only category slugs and IDs, category-to-URL mappings are grabbed from JavaScript on website. Therefore URLs were not guaranteed to be correct.Now all news are placed under/post/URL path, without category URL.Commandfetch-news-pagesFetch news pages contents for pages which were:From current sourceNot marked as "invalid URL" during previous fetchNot already fetchedExamplepythonnews_fetcher/prostoprosport_news_fetcher.pyfetch-news-pagesCommandgenerate-wiki-pagesGenerate MediaWiki pages as text files for fetched news pages not marked as uploaded.Options--output-file FILE— output JSON file with list of generated pages, it contains dictionary, where keys are page titles, and values are page file paths--output-directory FILE— directory to place generated MediaWiki page files--bot-name STRING— name of bot user account to use in page templateExamplepythonnews_fetcher/prostoprosport_news_fetcher.pygenerate-wiki-pages--output-file../data/pages.json--output-directory../data/pages/Commandmark-uploaded-pagesMark news articles as uploaded in database.Options--input-file FILEinput JSON file generated bygenerate-wiki-pagescommandExamplepythonnews_fetcher/prostoprosport_news_fetcher.pymark-uploaded-pages--input-file../data/pages.jsonModule-specific commandprocess-categoriesinprostoprosportsource moduleBuild categories mapping file. It will contain data about base URL for category slugs and IDs. For example, categoryrplhave base URL (without leading slash)football/russia/rpl.Options--input-from-js-file FILE— JSON file with categories data grabbed from JavaScript, default isdata/categories_from_js.json--input-bonus-file FILE— JSON file with additional data, default isdata/categories_bonus.json--input-colors-file FILE— JSON file with ID-to-color mapping data grabbed from JavaScript, default isdata/category_colors.json--output-file FILE— output JSON file, default isdata1/categories_data.jsonExamplepythonnews_fetcher/prostoprosport.pyprocess-categories