Package: newspaper4k
Newspaper4k: Article Scraping & Curation, a continuation of the beloved newspaper3k by codelucasAt the moment the Newspaper4k Project is a fork of the well known newspaper3k  bycodelucaswhich was not updated since Sept 2020. The initial goal of this fork is to keep the project alive and to add new features and fix bugs.I have duplicated all issues on the original project and will try to fix them. If you have any issues or feature requests please open an issue here.Python compatibility- Recommended: Python 3.8+
- Python 3.6+ minimum
- Fixes for Python < 3.8 are low priority and might not be mergedQuick startpipinstallnewspaper4kYou can start directly from the command line, using the included CLI:python-mnewspaper--url="https://edition.cnn.com/2023/11/17/success/job-seekers-use-ai/index.html"--language=en--output-format=json--output-file=article.jsonOr use the Python API:importnewspaperarticle=newspaper.article('https://edition.cnn.com/2023/10/29/sport/nfl-week-8-how-to-watch-spt-intl/index.html')print(article.authors)# ['Hannah Brewitt', 'Minute Read', 'Published', 'Am Edt', 'Sun October']print(article.publish_date)# 2023-10-29 09:00:15.717000+00:00print(article.text)# New England Patriots head coach Bill Belichick, right, embraces Buffalo Bills head coach Sean McDermott ...print(article.top_image)#https://media.cnn.com/api/v1/images/stellar/prod/231015223702-06-nfl-season-gallery-1015.jpg?c=16x9&q=w_800,c_fillprint(article.movies)# []article.nlp()print(article.keywords)# ['broncos', 'game', 'et', 'wide', 'chiefs', 'mahomes', 'patrick', 'denver', 'nfl', 'stadium', 'week', 'quarterback', 'win', 'history', 'images']print(article.summary)# Kevin Sabitus/Getty Images Denver Broncos running back Javonte Williams evades Green Bay Packers safety Darnell Savage, bottom.# Kathryn Riley/Getty Images Kansas City Chiefs quarterback Patrick Mahomes calls a play during the Chiefs' 19-8 Thursday Night Football win over the Denver Broncos on October 12.# Paul Sancya/AP New York Jets running back Breece Hall carries the ball during a game against the Denver Broncos.# The Broncos have not beaten the Chiefs since 2015, and have never beaten Chiefs quarterback Patrick Mahomes.# Australia: NFL+, ESPN, 7Plus Brazil: NFL+, ESPN Canada: NFL+, CTV, TSN, RDS Germany: NFL+, ProSieben MAXX, DAZN Mexico: NFL+, TUDN, ESPN, Fox Sports, Sky Sports UK: NFL+, Sky Sports, ITV, Channel 5 US: NFL+, CBS, NBC, FOX, ESPN, Amazon PrimeUsing the builder APIThis way you can build a Source object from a newspaper websites. This object will allow you to get all the articles and categories on the website. When you build the source, articles are not yet downloaded. You need to calldownload_articles()to download the articles, but note that it can take a significant time.```pythonimportnewspapercnn_paper=newspaper.build('http://cnn.com')print(cnn_paper.category_urls())# ['https://cnn.com', 'https://money.cnn.com', 'https://arabic.cnn.com', 'https://cnnespanol.cnn.com', 'http://edition.cnn.com', 'https://edition.cnn.com', 'https://us.cnn.com', 'https://www.cnn.com']article_urls=[article.urlforarticleincnn_paper.articles]print(article_urls[:3])# ['https://arabic.cnn.com/middle-east/article/2023/10/30/number-of-hostages-held-in-gaza-now-up-to-239-idf-spokesperson', 'https://arabic.cnn.com/middle-east/video/2023/10/30/v146619-sotu-sullivan-hostage-negotiations', 'https://arabic.cnn.com/middle-east/article/2023/10/29/norwegian-pm-israel-gaza']article=cnn_paper.articles[0]article.download()article.parse()print(article.title)# المتحدث باسم الجيش الإسرائيلي: عدد الرهائن المحتجزين في غزة يصل إلى```pyconfromnewspaperimportfulltexthtml=requests.get(...).texttext=fulltext(html)LanguagesNewspaper can extract and detect languagesseamlessly. If no language
is specified, Newspaper will attempt to auto detect a language from the available meta data. The fallback language is English.```pyconfromnewspaperimportArticlearticle=Article('https://www.bbc.com/zhongwen/simp/chinese-news-67084358')article.download()article.parse()print(article.title)# 晶片大战：台湾厂商助攻华为突破美国封锁？DocsCheck outThe Docsfor full and
detailed guides using newspaper.ContributingAdding languagesInterested in adding a new language for us? Refer to:Docs - Adding new
languagesAt the moment we are not integrating new languages, the language api will change.
You can still submit a PR with the language you want to add and we will merge it once the language api is stable.Submitting a PRInterested in submitting a PR? Refer to:Docs - Submitting a PRSubmitting an issueBefore submitting an issue, please check if it has already been reported. Additionally, please check that:The article website you have troubles with is not paywalledDocs - PaywallThe article website is not generating the webpage dynamically (e.g. using JavaScript)Docs - Dynamic contentThe article website is not using a language that is not supported by newspaper4kDocs - Supported languagesAlso, in any case, please provide the following information:The URL of the article you are trying to parseThe code you are using to parse the articleThe error you are getting (if any)The parsing result you are getting (if any)FeaturesMulti-threaded article download frameworkNews url identificationText extraction from htmlTop image extraction from htmlAll image extraction from htmlKeyword extraction from textSummary extraction from textAuthor extraction from textGoogle trending terms extractionWorks in 10+ languages (English, Chinese, German, Arabic, ...)Requirements and dependenciesFollowing system packages are required:PIL:libjpeg-devzlib1g-devlibpng12-devlxml:libxml2-devlibxslt-devPython Development version:python-devIf you are on Debian / Ubuntu, install using the following:Installpython3andpython3-dev:$ sudo apt-get install python3 python3-devInstallpip3command needed to installnewspaper4kpackage:$ sudo apt-get install python3-piplxml requirements:$ sudo apt-get install libxml2-dev libxslt-devFor PIL to recognize .jpg images:$ sudo apt-get install libjpeg-dev zlib1g-dev libpng12-devNOTE: If you find problem installinglibpng12-dev, try installinglibpng-dev.Install the distribution via pip:$ pip3 install newspaper4kDownload NLP (nltk) related corpora:$ curl https://raw.githubusercontent.com/AndyTheFactory/newspaper4k/master/download_corpora.py | python3Download NLP (nltk) related corpora:$ curl https://raw.githubusercontent.com/AndyTheFactory/newspaper4k/master/download_corpora.py | python3If you are on OSX, install using the following, you may use both
homebrew or macports:$ brew install libxml2 libxslt

$ brew install libtiff libjpeg webp little-cms2

$ pip3 install newspaper4k

$ curl https://raw.githubusercontent.com/AndyTheFactory/newspaper4k/master/download_corpora.py | python3LICENSEAuthored and maintained byAndrei Paraschiv.Newspaper was originally developed by Lucas Ou-Yang (codelucas), the original repository can be foundhere. Newspaper is licensed under the MIT license.CreditsThanks to Lucas Ou-Yang for creating the original Newspaper3k project and to all contributors of the original project.