Package: mediacloud-metadata
Media Cloud Metadata Extractor🚧under construction🚧This is a package to extract a domain, title, publication date, text, and language content from the URL or text of an
online news story. The methods for each are extracted from the largerMedia Cloud project,
but also build on numerous 3rd party  libraries. The metadata extracted includes:the original URL of publicationa normalized URL useful for de-duplicationthe canonical domain published onthe date of publicationthe primary language used in the article textthe title of the articlea normalized title useful for de-duplicationthe text content of the news articlethe name of the library used to extract the article contentInstallationpip install mediacloud-metadataUsageIf you pass in a URL, it will follow redirects and fetch the HTML for you.frommcmetadataimportextractmetadata=extract(url="https://my.awesome.news/story-path")You can also pass in HTML you already have on hand. Note that in this case it is also useful to pass in the URL
because that is used for some for some of the metadata extraction.frommcmetadataimportextractmetadata=extract(url="https://my.awesome.news/story-path",html_text="<html><head><title>my webpage ... </html>")DistributionPrep ReleaseRunpytestto make sure all the test passUpdate the version number inmcmetadata/__init__.pyMake a brief note in the version history section below about the changesCommit the changesTag the commit with a semantic version number - 'v*..'Push to repo to GitHubManual ReleaseRunpython setup.py sdistto create an installation packageRuntwine upload --repository-url https://test.pypi.org/legacy/ dist/*to upload it to PyPI's test platformRuntwine upload dist/*to upload it to PyPITest CacheTest are run against fixtures by default.  This can be changed with the use of '--use-cache=False' when running tests.
When adding new tests, re-run 'scripts/get-test-web-content.py'Version Historyv0.12.0: Add new const to centrally store MC User-Agent (mcmetadata.webpages.MEDIA_CLOUD_USER_AGENT)v0.11.2: Fix title parsing and url normalization edge cases, update requirementsv0.11.1: Add newurls.unique_url_hashhelper to centralize logic for generating a unique hash for a URL, also
returned fromextractin case you choose to use itv0.11.0: (error release)v0.10.0: Support defaults and overrides inextract, returning execution time stats, requirements updates, more
handling of malformed URLsv0.9.5: Updated requirements, update non-news site list, fix failing unit tests, tweak title parsing logicv0.9.4: Updated requirements to use faust-cchardet for py >3.9 supportv0.9.3: Updated content extractor dependencies, added py.typing for typing supportv0.9.2: fixed a bug related to title regex matchingv0.9.1: better support for some non-US government domainsv0.9.0: addsfeeds.normalize_urlhelperv0.8.2: small fix to url parsingv0.8.1: handle IP addresses in canonical_domain helperv0.8.0: update dependencies, fix various edge-case bugsv0.7.9: fixinclude_other_metadataprocessing, upgrade underlying libraries to latest, remove leading and
trailing whitespace from extracted textv0.7.8: add optionalinclude_other_metadataarg to extract method, which includes top_image and authors and
other less validated metadata in resultsv0.7.7: fix typov0.7.6: fix distribution packaging errorv0.7.5: add performance monitoring, handle invalid URLs, add a list of high volume non-news domains that might be
worth ignoring (based on high volume "noise" domains in our production database)v0.7.4: don't treat shortened URLs as homepage ones, also more aggressively strip URL query paramsv0.7.3: tweak title extraction for multipart titles, add is_homepage helper booleanv0.7.2: fix extraction argument bug introduced in last release, fix some more test casesv0.7.1: fix bug in url normalization, increase robustness in extractor chainv0.7.0: fix YouTube url normalization, better Trafilatura defaults, limit to pub dates within 90 days of today,
ensure language is 2 letters, content extraction performance improvements, fix some title parsing bugs,
add more test cases, add script to compare results to older Media Cloud code (which this stuff is
extracted from), resolve language guessing conflicts better, handle text encoding errorsv0.6.0: prefer language from metadata over guessing, try Trafilatura as first parser, encoding fixesv0.5.5: turn off aggressive date finding mode, which was making lots of 1/1 date guessesv0.5.4: bug in regex that parses og:title properties into titlesv0.5.3: bug fixes in title normalizationv0.5.2: more efficient parsing of dates from HTML, remove failing over-specified canonical domain casev0.5.1: fix small bug related to use of BeautifulSoupv0.5.0: add normalized URL and normalized titlev0.4.3:morework on title regex bugv0.4.2: work on title regex bugv0.4.1: work on deploymentv0.4.0: performance improvements, dependency updatesv0.3.1: update dependenciesv0.3.0: more fault tolerant, faster regex's, track extraction rates, update requirementsv0.2.0: first packaging release for use in other placesv0.1.1: first version for testing with collaboratorsContributorsCreated as part of the Media Cloud Project. Contributes include:Rahul Bhargava (Media Cloud, Northeastern University)Paige Gulley (Media Cloud)Phil Budne (Media Cloud)Vangelis Banos (Internet Archive)