Package: htmldate
Find original and updated publication dates of any web page. From the command-line or within Python, all the steps needed from web page download to HTML parsing, scraping, and text analysis are included.In a nutshellWith Python:>>>fromhtmldateimportfind_date>>>find_date('http://blog.python.org/2016/12/python-360-is-now-available.html')'2016-12-23'On the command-line:$htmldate-uhttp://blog.python.org/2016/12/python-360-is-now-available.html'2016-12-23'FeaturesMultilingual, robust and efficient (used in production on millions of documents)URLs, HTML files, or HTML trees are given as input (includes batch processing)Output as string in any date format (defaults toISO 8601 YMD)Detection of both original and updated datesCompatible with all recent versions of Pythonhtmldatecan examine markup and text. It provides the following ways to date an HTML document:Markup in header: Common patterns are used to identify relevant elements (e.g.linkandmetaelements) includingOpen Graph protocolattributesHTML code: The whole document is searched for structural markers:abbrortimeelements and a series of attributes (e.g.postmetadata)Bare HTML content: Heuristics are run on text and markup:infastmode the HTML page is cleaned and precise patterns are targetedinextensivemode all potential dates are collected and a disambiguation algorithm determines the best oneFinally the output is validated and converted to the chosen format.Performance1000 web pages containing identifiable dates (as of 2023-11-13 on Python 3.10)Python PackagePrecisionRecallAccuracyF-ScoreTimearticleDateExtractor 0.200.8030.7340.6220.7675xdate_guesser 2.1.40.7810.6000.5140.67918xgoose3 3.1.170.8690.5320.4930.66015xhtmldate[all] 1.6.0 (fast)0.8830.9240.8230.9031xhtmldate[all] 1.6.0 (extensive)0.8700.9930.8650.9281.7xnewspaper3k 0.2.80.7690.6670.5560.71515xnews-please 1.5.350.8010.7680.6450.78434xFor complete results and explanations see theevaluation page.InstallationThis Python package is tested on Linux, macOS and Windows systems; it is compatible with Python 3.6 upwards. It is available on the package repositoryPyPIand can notably be installed withpip(pip3where applicable):pip install htmldateand optionallypip install htmldate[speed].DocumentationFor more details on installation, Python & CLI usage,please refer to the documentation:htmldate.readthedocs.ioLicensehtmldateis distributed under theGNU General Public License v3.0. If you wish to redistribute this library but feel bounded by the license conditions please try interactingat arms length,multi-licensingwithcompatible licenses, orcontacting me.See alsoGPL and free software licensing: What’s in it for business?AuthorThis effort is part of methods to derive information from web documents in order to buildtext databases for research(chiefly linguistic analysis and natural language processing). Extracting and pre-processing web texts to the exacting standards of scientific research presents a substantial challenge for those who conduct such research. There are web pages for which neither the URL nor the server response provide a reliable way to find out when a document was published or modified. For more information:@article{barbaresi-2020-htmldate,title={{htmldate:APythonpackagetoextractpublicationdatesfromwebpages}},author="Barbaresi, Adrien",journal="Journal of Open Source Software",volume=5,number=51,pages=2439,url={https://doi.org/10.21105/joss.02439},publisher={TheOpenJournal},year=2020,}Barbaresi, A. “htmldate: A Python package to extract publication dates from web pages”, Journal of Open Source Software, 5(51), 2439, 2020. DOI: 10.21105/joss.02439Barbaresi, A. “Generic Web Content Extraction with Open-Source Software”, Proceedings of KONVENS 2019, Kaleidoscope Abstracts, 2019.Barbaresi, A. “Efficient construction of metadata-enhanced web corpora”, Proceedings of the10th Web as Corpus Workshop (WAC-X), 2016.You can contact me via mycontact pageorGitHub.ContributingContributionsare welcome!Feel free to file issues on thededicated page. Thanks to thecontributorswho submitted features and bugfixes!Kudos to the following software libraries:lxml,dateparserA few patterns are derived from thepython-goose,metascraper,newspaperandarticleDateExtractorlibraries. This module extends their coverage and robustness significantly.