Package: tldextract
tldextracttldextractaccurately separates a URL's subdomain, domain, and public suffix,
usingthe Public Suffix List (PSL).Say you want just the "google" part ofhttps://www.google.com.Everybody gets
this wrong.Splitting on the "." and taking the 2nd-to-last element only works
for simple domains, e.g. .com. Considerhttp://forums.bbc.co.uk: the naive splitting method
will give you "co" as the domain, instead of "bbc". Rather than juggle TLDs,
gTLDs, or ccTLDs  yourself,tldextractextracts the currently living public
suffixes according tothe Public Suffix List.A "public suffix" is one under which Internet users can directly register
names.A public suffix is also sometimes called an effective TLD (eTLD).Usage>>>importtldextract>>>tldextract.extract('http://forums.news.cnn.com/')ExtractResult(subdomain='forums.news',domain='cnn',suffix='com',is_private=False)>>>tldextract.extract('http://forums.bbc.co.uk/')# United KingdomExtractResult(subdomain='forums',domain='bbc',suffix='co.uk',is_private=False)>>>tldextract.extract('http://www.worldbank.org.kg/')# KyrgyzstanExtractResult(subdomain='www',domain='worldbank',suffix='org.kg',is_private=False)Note subdomain and suffix areoptional. Not all URL-like inputs have a
subdomain or a valid suffix.>>>tldextract.extract('google.com')ExtractResult(subdomain='',domain='google',suffix='com',is_private=False)>>>tldextract.extract('google.notavalidsuffix')ExtractResult(subdomain='google',domain='notavalidsuffix',suffix='',is_private=False)>>>tldextract.extract('http://127.0.0.1:8080/deployed/')ExtractResult(subdomain='',domain='127.0.0.1',suffix='',is_private=False)To rejoin the original hostname, if it was indeed a valid, registered hostname:>>>ext=tldextract.extract('http://forums.bbc.co.uk')>>>ext.registered_domain'bbc.co.uk'>>>ext.fqdn'forums.bbc.co.uk'By default, this package supports the public ICANN TLDs and their exceptions.
You can optionally support the Public Suffix List's private domains as well.This package started by implementing the chosen answer fromthis StackOverflow question on
getting the "domain name" from a URL.
However, the proposed regex solution doesn't address many country codes like
com.au, or the exceptions to country codes like the registered domain
parliament.uk. The Public Suffix List does, and so does this package.InstallLatest release on PyPI:pipinstalltldextractOr the latest dev version:pipinstall-e'git://github.com/john-kurkowski/tldextract.git#egg=tldextract'Command-line usage, splits the URL components by space:tldextracthttp://forums.bbc.co.uk# forums bbc co.ukNote about cachingBeware when first callingtldextract, it updates its TLD list with a live HTTP
request. This updated TLD set is usually cached indefinitely in$HOME/.cache/python-tldextract.
To control the cache's location, set TLDEXTRACT_CACHE environment variable or set the
cache_dir path in TLDExtract initialization.(Arguably runtime bootstrapping like that shouldn't be the default behavior,
like for production systems. But I want you to have the latest TLDs, especially
when I haven't kept this code up to date.)# extract callable that falls back to the included TLD snapshot, no live HTTP fetchingno_fetch_extract=tldextract.TLDExtract(suffix_list_urls=())no_fetch_extract('http://www.google.com')# extract callable that reads/writes the updated TLD set to a different pathcustom_cache_extract=tldextract.TLDExtract(cache_dir='/path/to/your/cache/')custom_cache_extract('http://www.google.com')# extract callable that doesn't use cachingno_cache_extract=tldextract.TLDExtract(cache_dir=None)no_cache_extract('http://www.google.com')If you want to stay fresh with the TLD definitions--though they don't change
often--delete the cache file occasionally, or runtldextract--updateor:envTLDEXTRACT_CACHE="~/tldextract.cache"tldextract--updateIt is also recommended to delete the file after upgrading this lib.Advanced usagePublic vs. private domainsThe PSLmaintains a concept of "private"
domains.PRIVATE domains are amendments submitted by the domain holder, as an
expression of how they operate their domain security policy. … While some
applications, such as browsers when considering cookie-setting, treat all
entries the same, other applications may wish to treat ICANN domains and
PRIVATE domains differently.By default,tldextracttreats public and private domains the same.>>>extract=tldextract.TLDExtract()>>>extract('waiterrant.blogspot.com')ExtractResult(subdomain='waiterrant',domain='blogspot',suffix='com',is_private=False)The following overrides this.>>>extract=tldextract.TLDExtract()>>>extract('waiterrant.blogspot.com',include_psl_private_domains=True)ExtractResult(subdomain='',domain='waiterrant',suffix='blogspot.com',is_private=True)or to change the default for all extract calls,>>>extract=tldextract.TLDExtract(include_psl_private_domains=True)>>>extract('waiterrant.blogspot.com')ExtractResult(subdomain='',domain='waiterrant',suffix='blogspot.com',is_private=True)The thinking behind the default is, it's the more common case when people
mentally parse a domain name. It doesn't assume familiarity with the PSL nor
that the PSL makes a public/private distinction. Note this default may run
counter to the default parsing behavior of other, PSL-based libraries.Specifying your own URL or file for Public Suffix List dataYou can specify your own input data in place of the default Mozilla Public Suffix List:extract=tldextract.TLDExtract(suffix_list_urls=["http://foo.bar.baz"],# Recommended: Specify your own cache file, to minimize ambiguities about where# tldextract is getting its data, or cached data, from.cache_dir='/path/to/your/cache/',fallback_to_snapshot=False)The above snippet will fetch from the URLyouspecified, upon first need to download the
suffix list (i.e. if the cached version doesn't exist).If you want to use input data from your local filesystem, just use thefile://protocol:extract=tldextract.TLDExtract(suffix_list_urls=["file://"+"/absolute/path/to/your/local/suffix/list/file"],cache_dir='/path/to/your/cache/',fallback_to_snapshot=False)Use an absolute path when specifying thesuffix_list_urlskeyword argument.os.pathis your friend.The command line update command can be used with a URL or local file you specify:tldextract--update--suffix_list_url"http://foo.bar.baz"This could be useful in production when you don't want the delay associated with updating the suffix
list on first use, or if you are behind a complex firewall that prevents a simple update from working.FAQCan you add suffix ____? Can you make an exception for domain ____?This project doesn't contain an actual list of public suffixes. That comes fromthe Public Suffix List (PSL). Submit amendments there.(In the meantime, you can tell tldextract about your exception by either
forking the PSL and using your fork in thesuffix_list_urlsparam, or adding
your suffix piecemeal with theextra_suffixesparam.)I see my suffix inthe Public Suffix List (PSL), but this library doesn't extract it.Check if your suffix is in the private section of the list. Seethis
documentation.If I pass an invalid URL, I still get a result, no error. What gives?To keeptldextractlight in LoC & overhead, and because there are plenty of
URL validators out there, this library is very lenient with input. If valid
URLs are important to you, validate them before callingtldextract.To avoid parsing a string twice, you can passtldextractthe output ofurllib.parsemethods.
For example:extractor=TLDExtract()split_url=urllib.parse.urlsplit("https://foo.bar.com:8080")split_suffix=extractor.extract_urllib(split_url)url_to_crawl=f"{split_url.scheme}://{split_suffix.registered_domain}:{split_url.port}"tldextract's lenient string parsing stance lowers the learning curve of using
the library, at the cost of desensitizing users to the nuances of URLs. This
could be overhauled. For example, users could opt into validation, either
receiving exceptions or error metadata on results.ContributeSetting upgit clonethis repository.Change into the new directory.pip install --upgrade --editable '.[testing]'Running the test suiteRun all tests against all supported Python versions:tox--parallelRun all tests against a specific Python environment configuration:tox-l
tox-epy311Code StyleAutomatically format all code:black.