Package: ural
UralA python helper library full of URL-related heuristics.InstallationYou can installuralwith pip with the following command:pip install uralHow to cite?uralis published onZenodoasYou can cite it thusly:Guillaume Plique, Jules Farjas, Oubine Perrin, Benjamin Ooghe-Tabanou, Martin Delabre, Pauline Breteau, Jean Descamps, Béatrice Mazoyer, Amélie Pellé, Laura Miguel, & César Pichon. Ural, a python helper library full of URL-related heuristics. (2018). Zenodo.https://doi.org/10.5281/zenodo.8160139UsageGeneric functionscanonicalize_urlcould_be_htmlcould_be_rssensure_protocolfingerprint_hostnamefingerprint_urlforce_protocolformat_urlget_domain_nameget_hostnameget_fingerprinted_hostnameget_normalized_hostnamehas_special_hosthas_valid_suffixhas_valid_tldinfer_redirectionis_homepageis_shortened_urlis_special_hostis_typo_urlis_urlis_valid_tldlinks_from_htmlnormalize_hostnamenormalize_urlshould_follow_hrefshould_resolvesplit_suffixstrip_protocolurlpathspliturls_from_htmlurls_from_textUtilitiesUpgrading suffixes and TLDsClassesHostnameTrieSet#.add#.matchLRU-related functions(What on earth is a LRU?)lru.url_to_lrulru.lru_to_urllru.lru_stemslru.canonicalized_lru_stemslru.normalized_lru_stemslru.fingerprinted_lru_stemslru.serialize_lrulru.unserialize_lruLRU-related classesLRUTrie#.set#.set_lru#.match#.match_lruCanonicalizedLRUTrieNormalizedLRUTrieFingerprintedLRUTriePlatform-specific functionsfacebookhas_facebook_commentsis_facebook_idis_facebook_full_idis_facebook_urlis_facebook_post_urlis_facebook_linkconvert_facebook_url_to_mobileparse_facebook_urlextract_url_from_facebook_linkgoogleis_amp_urlis_google_linkextract_url_from_google_linkextract_id_from_google_drive_urlparse_google_drive_urlinstagramis_instagram_post_shortcodeis_instagram_usernameis_instagram_urlextract_username_from_instagram_urlparse_instagram_urltelegramis_telegram_message_idis_telegram_urlconvert_telegram_url_to_publicextract_channel_name_from_telegram_urlparse_telegram_urltwitteris_twitter_urlextract_screen_name_from_twitter_urlparse_twitter_urlyoutubeis_youtube_urlis_youtube_channel_idis_youtube_video_idparse_youtube_urlextract_video_id_from_youtube_urlnormalize_youtube_urlDifferences between canonicalize_url, normalize_url & fingerprint_urluralcomes with three different url deduplication schemes, targeted to different use-cases and ordered hereafter by ascending aggressiveness:canonicalize_url: we clean the url by performing some light preprocessing usually done by web browsers before hitting them, e.g. lowercasing the hostname, decoding punycode, ensuring we have a protocol, dropping leading and trailing whitespace etc. The clean url is guaranteed to still lead to the same place.normalize_url: we apply more advanced preprocessing that will drop some parts of the url that are irrelevant to where the url leads, such as technical artifacts and SEO tricks. For instance, we will drop typical query items used by marketing campaigns, reorder the query items, infer some redirections, strip trailing slash or fragment when advisable etc. At that point, the url should be clean enough that one can perform meaningful statistical aggregation when counting them, all while ensuring with some good probability that the url still works and still leads to the same place, at least if the target server follows most common conventions.fingerprint_url: we go a step further and we perform destructive preprocessing that cannot guarantee that the resulting url will still be valid. But the result might be even more useful for statistical aggregation, especially when counting urls from large platforms having multiple domains (e.g.facebook.com,facebook.fretc.)FunctionUse-casesUrl validityDeduplication strengthcanonicalize_urlweb crawlerTechnically the same+normalize_urlweb crawler, statistical aggregationProbably the same++fingerprint_urlstatistical aggregationPotentially invalid+++Examplefromuralimportcanonicalize_url,normalize_url,fingerprint_urlurl='https://www.FACEBOOK.COM:80/index.html?utc_campaign=3&id=34'canonicalize_url(url)>>>'https://www.facebook.com/index.html?utc_campaign=3&id=34'# The same url, cleaned up a littlenormalize_url(url)>>>'facebook.com?id=34'# Still a valid url, with implicit protocol, where all the cruft has been discardedfingerprint_url(url,strip_suffix=True)>>>'facebook?id=34'# Not a valid url anymore, but useful to match more potential# candidates such as: http://facebook.co.uk/index.html?id=34canonicalize_urlFunction returning a clean and safe version of the url by performing the same kind of preprocessing as web browsers.For more details about this be sure to readthissection of the docs.fromuralimportcanonicalize_urlcanonicalize_url('www.LEMONDE.fr')>>>'https://lemonde.fr'Argumentsurlstring: url to canonicalize.quoted?bool[False]: by default the function will unquote the url as much as possible all while keeping the url safe. If this kwarg is set toTrue, the function will instead quote the url as much as possible all while ensuring nothing will be double-quoted.default_protocol?str[https]: default protocol to add when the url has none.strip_fragment?str[False]: whether to strip the url's fragment.could_be_htmlFunction returning whether the url could return HTML.fromuralimportcould_be_htmlcould_be_html('https://www.lemonde.fr')>>>Truecould_be_html('https://www.lemonde.fr/articles/page.php')>>>Truecould_be_html('https://www.lemonde.fr/data.json')>>>Falsecould_be_html('https://www.lemonde.fr/img/figure.jpg')>>>Falsecould_be_rssFunction returning whether the given url could be a rss feed url.fromuralimportcould_be_rsscould_be_rss('https://www.lemonde.fr/cyclisme/rss_full.xml')>>>Truecould_be_rss('https://www.lemonde.fr/cyclisme/')>>>Falsecould_be_rss('https://www.ecorce.org/spip.php?page=backend')>>>Truecould_be_rss('https://feeds.feedburner.com/helloworld')>>>Trueensure_protocolFunction checking if the url has a protocol, and adding the given one if there is none.fromuralimportensure_protocolensure_protocol('www.lemonde.fr',protocol='https')>>>'https://www.lemonde.fr'Argumentsurlstring: URL to format.protocolstring: protocol to use if there is none inurl. Is 'http' by default.fingerprint_hostnameFunction returning a "fingerprinted" version of the given hostname by stripping subdomains irrelevant for statistical aggregation. Be warned that this function is even more aggressive thannormalize_hostnameand that the resulting hostname might not be valid anymore.For more details about this be sure to readthissection of the docs.fromuralimportfingerprint_hostnamefingerprint_hostname('www.lemonde.fr')>>>'lemonde.fr'fingerprint_hostname('fr-FR.facebook.com')>>>'facebook.com'fingerprint_hostname('fr-FR.facebook.com',strip_suffix=True)>>>'facebook'Argumentshostnamestring: target hostname.strip_suffix?bool[False]: whether to strip the hostname suffix such as.comor.co.uk. This can be useful to aggegate different domains of the same platform.fingerprint_urlFunction returning a "fingerprinted" version of the given url that can be useful for statistical aggregation. Be warned that this function is even more aggressive thannormalize_urland that the resulting url might not be valid anymore.For more details about this be sure to readthissection of the docs.fromuralimportfingerprint_hostnamefingerprint_url('www.lemonde.fr/article.html')>>>'lemonde.fr/article.html'fingerprint_url('fr-FR.facebook.com/article.html')>>>'facebook.com/article.html'fingerprint_url('fr-FR.facebook.com/article.html',strip_suffix=True)>>>'facebook/article.html'Argumentsurlstring: target url.strip_suffix?bool[False]: whether to strip the hostname suffix such as.comor.co.uk. This can be useful to aggegate different domains of the same platform.platform_aware?bool[False]: whether to take some well-known platforms supported byuralsuch as facebook, youtube etc. into account when normalizing the url.force_protocolFunction force-replacing the protocol of the given url.fromuralimportforce_protocolforce_protocol('https://www2.lemonde.fr',protocol='ftp')>>>'ftp://www2.lemonde.fr'Argumentsurlstring: URL to format.protocolstring: protocol wanted in the output url. Is'http'by default.format_urlFunction formatting a url given some typical parameters.fromuralimportformat_urlformat_url('https://lemonde.fr',path='/article.html',args={'id':'48675'},fragment='title-2')>>>'https://lemonde.fr/article.html?id=48675#title-2'# Path can be given as an iterableformat_url('https://lemonde.fr',path=['articles','one.html'])>>>'https://lemonde.fr/articles/one.html'# Extensionformat_url('https://lemonde.fr',path=['article'],ext='html')>>>'https://lemonde.fr/articles/article.html'# Query args are formatted/quoted and/or skipped if None/Falseformat_url("http://lemonde.fr",path=["business","articles"],args={"hello":"world","number":14,"boolean":True,"skipped":None,"also-skipped":False,"quoted":"test=ok",},fragment="#test",)>>>'http://lemonde.fr/business/articles?boolean&hello=world&number=14&quoted=test%3Dok#test'# Custom argument value formattingdefformat_arg_value(key,value):ifkey=='ids':return','.join(value)returnkeyformat_url('https://lemonde.fr',args={'ids':[1,2]},format_arg_value=format_arg_value)>>>'https://lemonde.fr?ids=1%2C2'# Formatter classfromuralimportURLFormatterformatter=URLFormatter('https://lemonde.fr',args={'id':'one'})formatter(path='/article.html')>>>'https://lemonde.fr/article.html?id=one'# same as:formatter.format(path='/article.html')>>>'https://lemonde.fr/article.html?id=one'# Query arguments are mergedformatter(path='/article.html',args={"user_id":"two"})>>>'https://lemonde.fr/article.html?id=one&user_id=two'# Easy subclassingclassMyCustomFormatter(URLFormatter):BASE_URL='https://lemonde.fr/api'defformat_api_call(self,token):returnself.format(args={'token':token})formatter=MyCustomFormatter()formatter.format_api_call('2764753')>>>'https://lemonde.fr/api?token=2764753'Argumentsbase_urlstr: Base url.path?str|list: the url's path.args?dict: query arguments as a dictionary.format_arg_value?callable: function taking a query argument key and value and returning the formatted value.fragment?str: the url's fragment.ext?str: path extension such as.html.get_domain_nameFunction returning an url's domain name. This function is of course tld-aware and will returnNoneif no valid domain name can be found.fromuralimportget_domain_nameget_domain_name('https://facebook.com/path')>>>'facebook.com'get_hostnameFunction returning the given url's full hostname. It can work on scheme-less urls.fromuralimportget_hostnameget_hostname('http://www.facebook.com/path')>>>'www.facebook.com'get_fingerprinted_hostnameFunction returning the "fingerprinted" hostname of the given url by stripping subdomains irrelevant for statistical aggregation. Be warned that this function is even more aggressive thanget_normalized_hostnameand that the resulting hostname might not be valid anymore.For more details about this be sure to readthissection of the docs.fromuralimportget_normalized_hostnameget_normalized_hostname('https://www.lemonde.fr/article.html')>>>'lemonde.fr'get_normalized_hostname('https://fr-FR.facebook.com/article.html')>>>'facebook.com'get_normalized_hostname('https://fr-FR.facebook.com/article.html',strip_suffix=True)>>>'facebook'Argumentsurlstring: target url.strip_suffix?bool[False]: whether to strip the hostname suffix such as.comor.co.uk. This can be useful to aggegate different domains of the same platform.get_normalized_hostnameFunction returning the given url's normalized hostname, i.e. without usually irrelevant subdomains etc. Works a lot likenormalize_url.For more details about this be sure to readthissection of the docs.fromuralimportget_normalized_hostnameget_normalized_hostname('http://www.facebook.com/path')>>>'facebook.com'get_normalized_hostname('http://fr-FR.facebook.com/path')>>>'facebook.com'Argumentsurlstr: Target url.infer_redirectionbool[True]: whether to attempt resolving common redirects by leveraging well-known GET parameters.normalize_amp?bool[True]: Whether to attempt to normalize Google AMP subdomains.has_special_hostFunction returning whether the given url looks like it has a special host.fromuralimporthas_special_hosthas_special_host('http://104.19.154.83')>>>Truehas_special_host('http://youtube.com')>>>Falsehas_valid_suffixFunction returning whether the given url has a valid suffix as perMozzila's Public Suffix List.fromuralimporthas_valid_suffixhas_valid_suffix('http://lemonde.fr')>>>Truehas_valid_suffix('http://lemonde.doesnotexist')>>>False# Also works with hostnameshas_valid_suffix('lemonde.fr')>>>Truehas_valid_tldFunction returning whether the given url has a valid Top Level Domain (TLD) as perIANA's list.fromuralimporthas_valid_tldhas_valid_tld('http://lemonde.fr')>>>Truehas_valid_tld('http://lemonde.doesnotexist')>>>False# Also works with hostnameshas_valid_tld('lemonde.fr')>>>Trueinfer_redirectionFunction attempting to find obvious clues in the given url that it is in fact a redirection and resolving the redirection automatically without firing any HTTP request. If nothing is found, the given url will be returned as-is.The function is by default recursive and will attempt to infer redirections until none is found, but you can disable this behavior if you need to.fromuralimportinfer_redirectioninfer_redirection('https://www.google.com/url?sa=t&source=web&rct=j&url=https%3A%2F%2Fm.youtube.com%2Fwatch%3Fv%3D4iJBsjHMviQ&ved=2ahUKEwiBm-TO3OvkAhUnA2MBHQRPAR4QwqsBMAB6BAgDEAQ&usg=AOvVaw0i7y2_fEy3nwwdIZyo_qug')>>>'https://m.youtube.com/watch?v=4iJBsjHMviQ'infer_redirection('https://test.com?url=http%3A%2F%2Flemonde.fr%3Fnext%3Dhttp%253A%252F%252Ftarget.fr')>>>'http://target.fr'infer_redirection('https://test.com?url=http%3A%2F%2Flemonde.fr%3Fnext%3Dhttp%253A%252F%252Ftarget.fr',recursive=False)>>>'http://lemonde.fr?next=http%3A%2F%2Ftarget.fr'is_homepageFunction returning whether the given url isprobablya website's homepage, based on its path.fromuralimportis_homepageis_homepage('http://lemonde.fr')>>>Trueis_homepage('http://lemonde.fr/index.html')>>>Trueis_homepage('http://lemonde.fr/business/article5.html')>>>Falseis_shortened_urlFunction returning whether the given url isprobablya shortened url. It works by matching the given url domain against most prominent shortener domains. So the result could be a false negative.fromuralimportis_shortened_urlis_shortened_url('http://lemonde.fr')>>>Falseis_shortened_url('http://bit.ly/1sNZMwL')>>>Trueis_special_hostFunction returning whether the given hostname looks like a special host.fromuralimportis_special_hostis_special_host('104.19.154.83')>>>Trueis_special_host('youtube.com')>>>Falseis_typo_urlFunction returning whether the given string isprobablya typo error. This function doesn't test if the given string is a valid url. It works by matching the given url tld against most prominent typo-like tlds or by matching the given string against most prominent inclusive language terminations. So the result could be a false negative.fromuralimportis_typo_urlis_typo_url('http://dirigeants.es')>>>Trueis_typo_url('https://www.instagram.com')>>>Falseis_urlFunction returning whether the given string is a valid url.fromuralimportis_urlis_url('https://www2.lemonde.fr')>>>Trueis_url('lemonde.fr/economie/article.php',require_protocol=False)>>>Trueis_url('lemonde.falsetld/whatever.html',tld_aware=True)>>>FalseArgumentsstringstring: string to test.require_protocolbool[True]: whether the argument has to have a protocol to be considered a url.tld_awarebool[False]: whether to check if the url's tld actually exists or not.allow_spaces_in_pathbool[False]: whether the allow spaces in URL paths.only_http_httpsbool[True]: whether to only allow thehttpandhttpsprotocols.is_valid_tldFunction returning whether the given Top Level Domain (TLD) is valid as perIANA's list.fromuralimportis_valid_tldis_valid_tld('.fr')>>>Trueis_valid_tld('com')>>>Trueis_valid_tld('.doesnotexist')>>>Falselinks_from_htmlFunction returning an iterator over the valid outgoing links present in given HTML text.This is a variant ofurls_from_htmlsuited to web crawlers. It can deduplicate the urls, canonicalize them, join them with a base url and filter out things that should not be followed such asmailto:orjavascript:href links etc. It will also skip any url equivalent to the given base url.Note this function is able to work both on string and bytes seamlessly.fromuralimportlinks_from_htmlhtml=b"""<p>Hey! Check this site:<a href="https://medialab.sciencespo.fr/">médialab</a>And also this page:<a href="article.html">article</a>Or click on this:<a href="javascript:alert('hello');">link</a></p>"""forlinkinlinks_from_html('http://lemonde.fr',html):print(link)>>>'https://medialab.sciencespo.fr/'>>>'http://lemonde.fr/article.html'Argumentsbase_urlstring: the HTML's url.stringstring|bytes: html string or bytes.encoding?string[utf-8]: if given binary, this encoding will be used to decode the found urls.canonicalize?bool[False]: whether to canonicalize the urls usingcanonicalize_url.strip_fragment?bool[False]: whether to strip the url fragments when usingcanonicalize.unique?bool[False]: whether to deduplicate the urls.normalize_hostnameFunction normalizing the given hostname, i.e. without usually irrelevant subdomains etc. Works a lot likenormalize_url.For more details about this be sure to readthissection of the docs.fromuralimportnormalize_hostnamenormalize_hostname('www.facebook.com')>>>'facebook.com'normalize_hostname('fr-FR.facebook.com')>>>'facebook.com'normalize_urlFunction normalizing the given url by stripping it of usually non-discriminant parts such as irrelevant query items or sub-domains etc.This is a very useful utility when attempting to match similar urls written slightly differently when shared on social media etc.For more details about this be sure to readthissection of the docs.fromuralimportnormalize_urlnormalize_url('https://www2.lemonde.fr/index.php?utm_source=google')>>>'lemonde.fr'Argumentsurlstring: URL to normalize.infer_redirection?bool[True]: whether to attempt resolving common redirects by leveraging well-known GET parameters.fix_common_mistakes?bool[True]: whether to attempt to fix common URL mistakes.normalize_amp?bool[True]: whether to attempt to normalize Google AMP urls.sort_query?bool[True]: whether to sort query items.strip_authentication?bool[True]: whether to strip authentication.strip_fragment?bool|str['except-routing']: whether to strip the url's fragment. If set toexcept-routing, will only strip the fragment if the fragment is not deemed to be js routing (i.e. if it contains a/).strip_index?bool[True]: whether to strip trailing index.strip_irrelevant_subdomains?bool[False]: whether to strip irrelevant subdomains such aswwwetc.strip_protocol?bool[True]: whether to strip the url's protocol.strip_trailing_slash?bool[True]: whether to strip trailing slash.quoted?bool[False]: by default the function will unquote the url as much as possible all while keeping the url safe. If this kwarg is set toTrue, the function will instead quote the url as much as possible all while ensuring nothing will be double-quoted.platform_aware?bool[False]: whether to take some well-known platforms supported byuralsuch as facebook, youtube etc. into account when normalizing the url.should_follow_hrefFunction returning whether the given href should be followed (usually from a crawler's context). This means it will filter out anchors, and url having a scheme which is not http/https.fromuralimportshould_follow_hrefshould_follow_href('#top')>>>Falseshould_follow_href('http://lemonde.fr')>>>Trueshould_follow_href('/article.html')>>>Trueshould_resolveFunction returning whether the given function looks like something you would want to resolve because the url willprobablylead to some redirection.It is quite similar tois_shortened_urlbut covers more ground since it also deal with url patterns which are not shortened per se.fromuralimportshould_resolveshould_resolve('http://lemonde.fr')>>>Falseshould_resolve('http://bit.ly/1sNZMwL')>>>Trueshould_resolve('https://doi.org/10.4000/vertigo.26405')>>>Truesplit_suffixFunction splitting a hostname or a url's hostname into the domain part and the suffix part (while respectingMozzila's Public Suffix List).fromuralimportsplit_suffixsplit_suffix('http://www.bbc.co.uk/article.html')>>>('www.bbc','co.uk')split_suffix('http://www.bbc.idontexist')>>>Nonesplit_suffix('lemonde.fr')>>>('lemonde','fr')strip_protocolFunction removing the protocol from the url.fromuralimportstrip_protocolstrip_protocol('https://www2.lemonde.fr/index.php')>>>'www2.lemonde.fr/index.php'Argumentsurlstring: URL to format.urlpathsplitFunction taking a url and returning its path, tokenized as a list.fromuralimporturlpathspliturlpathsplit('http://lemonde.fr/section/article.html')>>>['section','article.html']urlpathsplit('http://lemonde.fr/')>>>[]# If you want to split a path directlyfromuralimportpathsplitpathsplit('/section/articles/')>>>['section','articles']urls_from_htmlFunction returning an iterator over the urls present in the links of given HTML text.Note this function is able to work both on string and bytes seamlessly.fromuralimporturls_from_htmlhtml="""<p>Hey! Check this site: <a href="https://medialab.sciencespo.fr/">médialab</a></p>"""forurlinurls_from_html(html):print(url)>>>'https://medialab.sciencespo.fr/'Argumentsstringstring|bytes: html string or bytes.encoding?string[utf-8]: if given binary, this encoding will be used to decode the found urls.errors?string[strict]: policy on decode errors.urls_from_textFunction returning an iterator over the urls present in the string argument. Extracts only urls having a protocol.Note that this function is somewhat markdown-aware, and punctuation-aware.fromuralimporturls_from_texttext="Hey! Check this site: https://medialab.sciencespo.fr/, it looks really cool. They're developing many tools on https://github.com/"forurlinurls_from_text(text):print(url)>>>'https://medialab.sciencespo.fr/'>>>'https://github.com/'Argumentsstringstring: source string.Upgrading suffixes and TLDsIf you want to upgrade the package's data wrt Mozilla suffixes and IANA TLDs, you can do so either by running the following command:python-muralupgradeor directly in your python code:fromural.tldimportupgradeupgrade()# Or if you want to patch runtime only this time, or regularly# (for long running programs or to avoid rights issues etc.):upgrade(transient=True)HostnameTrieSetClass implementing a hierarchic set of hostnames so you can efficiently query whether urls match hostnames in the set.fromuralimportHostnameTrieSettrie=HostnameTrieSet()trie.add('lemonde.fr')trie.add('business.lefigaro.fr')trie.match('https://liberation.fr/article1.html')>>>Falsetrie.match('https://lemonde.fr/article1.html')>>>Truetrie.match('https://www.lemonde.fr/article1.html')>>>Truetrie.match('https://lefigaro.fr/article1.html')>>>Falsetrie.match('https://business.lefigaro.fr/article1.html')>>>True#.addMethod add a single hostname to the set.fromuralimportHostnameTrieSettrie=HostnameTrieSet()trie.add('lemonde.fr')Argumentshostnamestring: hostname to add to the set.#.matchMethod returning whether the given url matches any of the set's hostnames.fromuralimportHostnameTrieSettrie=HostnameTrieSet()trie.add('lemonde.fr')trie.match('https://liberation.fr/article1.html')>>>Falsetrie.match('https://lemonde.fr/article1.html')>>>TrueArgumentsurlstring|urllib.parse.SplitResult: url to match.lru.url_to_lruFunction converting the given url to a serialized lru.fromural.lruimporturl_to_lruurl_to_lru('http://www.lemonde.fr:8000/article/1234/index.html?field=value#2')>>>'s:http|t:8000|h:fr|h:lemonde|h:www|p:article|p:1234|p:index.html|q:field=value|f:2|'Argumentsurlstring: url to convert.suffix_aware?bool: whether to be mindful of suffixes when converting (e.g. considering "co.uk" as a single token).lru.lru_to_urlFunction converting the given serialized lru or lru stems to a proper url.fromural.lruimportlru_to_urllru_to_url('s:http|t:8000|h:fr|h:lemonde|h:www|p:article|p:1234|p:index.html|')>>>'http://www.lemonde.fr:8000/article/1234/index.html'lru_to_url(['s:http','h:fr','h:lemonde','h:www','p:article','p:1234','p:index.html'])>>>'http://www.lemonde.fr:8000/article/1234/index.html'lru.lru_stemsFunction returning url parts in hierarchical order.fromural.lruimportlru_stemslru_stems('http://www.lemonde.fr:8000/article/1234/index.html?field=value#2')>>>['s:http','t:8000','h:fr','h:lemonde','h:www','p:article','p:1234','p:index.html','q:field=value','f:2']Argumentsurlstring: URL to parse.suffix_aware?bool: whether to be mindful of suffixes when converting (e.g. considering "co.uk" as a single token).lru.canonicalized_lru_stemsFunction canonicalizing the url and returning its parts in hierarchical order.fromural.lruimportcanonicalized_lru_stemscanonicalized_lru_stems('http://www.lemonde.fr/article/1234/index.html?field=value#2')>>>['s:http','h:fr','h:lemonde','p:article','p:1234','q:field=value','f:2']ArgumentsThis function accepts the same arguments ascanonicalize_url.lru.normalized_lru_stemsFunction normalizing the url and returning its parts in hierarchical order.fromural.lruimportnormalized_lru_stemsnormalized_lru_stems('http://www.lemonde.fr/article/1234/index.html?field=value#2')>>>['h:fr','h:lemonde','p:article','p:1234','q:field=value']ArgumentsThis function accepts the same arguments asnormalize_url.lru.fingerprinted_lru_stemsFunction fingerprinting the url and returning its parts in hierarchical order.fromural.lruimportfingerprinted_lru_stemsfingerprinted_lru_stems('http://www.lemonde.fr/article/1234/index.html?field=value#2',strip_suffix=True)>>>['h:lemonde','p:article','p:1234','q:field=value']ArgumentsThis function accepts the same arguments asfingerprint_url.lru.serialize_lruFunction serializing lru stems to a string.fromural.lruimportserialize_lruserialize_lru(['s:https','h:fr','h:lemonde'])>>>'s:https|h:fr|h:lemonde|'lru.unserialize_lruFunction unserializing stringified lru to a list of stems.fromural.lruimportunserialize_lruunserialize_lru('s:https|h:fr|h:lemonde|')>>>['s:https','h:fr','h:lemonde']LRUTrieClass implementing a prefix tree (Trie) storing URLs hierarchically by storing them as LRUs along with some arbitrary metadata. It is very useful when needing to match URLs by longest common prefix.Note that this class directly inherits from thephylacterylibrary'sTrieDictso you can also use any of its methods.fromural.lruimportLRUTrietrie=LRUTrie()# To respect suffixestrie=LRUTrie(suffix_aware=True)#.setMethod storing a URL in a LRUTrie along with its metadata.fromural.lruimportLRUTrietrie=LRUTrie()trie.set('http://www.lemonde.fr',{'type':'general press'})trie.match('http://www.lemonde.fr')>>>{'type':'general press'}Argumentsurlstring: url to store in the LRUTrie.metadataany: metadata of the url.#.set_lruMethod storing a URL already represented as a LRU or LRU stems along with its metadata.fromural.lruimportLRUTrietrie=LRUTrie()# Using stemstrie.set_lru(['s:http','h:fr','h:lemonde','h:www'],{'type':'general press'})# Using serialized lrutrie.set_lru('s:http|h:fr|h:lemonde|h:www|',{'type':'general_press'})Argumentslrustring|list: lru to store in the Trie.metadataany: metadata to attach to the lru.#.matchMethod returning the metadata attached to the longest prefix match of your query URL. Will returnNoneif no common prefix can be found.fromural.lruimportLRUTrietrie=LRUTrie()trie.set('http://www.lemonde.fr',{'media':'lemonde'})trie.match('http://www.lemonde.fr')>>>{'media':'lemonde'}trie.match('http://www.lemonde.fr/politique')>>>{'media':'lemonde'}trie.match('http://www.lefigaro.fr')>>>NoneArgumentsurlstring: url to match in the LRUTrie.#.match_lruMethod returning the metadata attached to the longest prefix match of your query LRU. Will returnNoneif no common prefix can be found.fromural.lruimportLRUTrietrie=LRUTrie()trie.set(['s:http','h:fr','h:lemonde','h:www'],{'media':'lemonde'})trie.match(['s:http','h:fr','h:lemonde','h:www'])>>>{'media':'lemonde'}trie.match('s:http|h:fr|h:lemonde|h:www|p:politique|')>>>{'media':'lemonde'}trie.match(['s:http','h:fr','h:lefigaro','h:www'])>>>NoneArgumentslrustring|list: lru to match in the LRUTrie.CanonicalizedLRUTrieTheCanonicalizedLRUTrieis nearly identical to the standardLRUTrieexcept that it canonicalizes given urls before attempting any operation using thecanonicalize_urlfunction.Its constructor therefore takes the same arguments as the beforementioned function.fromural.lruimportCanonicalizedLRUTrietrie=CanonicalizedLRUTrie(strip_fragment=False)NormalizedLRUTrieTheNormalizedLRUTrieis nearly identical to the standardLRUTrieexcept that it normalizes given urls before attempting any operation using thenormalize_urlfunction.Its constructor therefore takes the same arguments as the beforementioned function.fromural.lruimportNormalizedLRUTrietrie=NormalizedLRUTrie(normalize_amp=False)FingerprintedLRUTrieTheFingerprintedLRUTrieis nearly identical to the standardLRUTrieexcept that it fingerprints given urls before attempting any operation using thefingerprint_urlfunction.Its constructor therefore takes the same arguments as the beforementioned function.fromural.lruimportFingerprintedLRUTrietrie=FingerprintedLRUTrie(strip_suffix=False)Facebookhas_facebook_commentsFunction returning whether the given url is pointing to a Facebook resource potentially having comments (such as a post, photo or video for instance).fromural.facebookimporthas_facebook_commentshas_facebook_comments('https://www.facebook.com/permalink.php?story_fbid=1354978971282622&id=598338556946671')>>>Truehas_facebook_comments('https://www.facebook.com/108824017345866/videos/311658803718223')>>>Truehas_facebook_comments('https://www.facebook.com/astucerie/')>>>Falsehas_facebook_comments('https://www.lemonde.fr')>>>Falsehas_facebook_comments('/permalink.php?story_fbid=1354978971282622&id=598338556946671',allow_relative_urls=True)>>>Trueis_facebook_idFunction returning whether the given string is a valid Facebook id or not.fromural.facebookimportis_facebook_idis_facebook_id('974583586343')>>>Trueis_facebook_id('whatever')>>>Falseis_facebook_full_idFunction returning whether the given string is a valid Facebook full post id or not.fromural.facebookimportis_facebook_full_idis_facebook_full_id('974583586343_9749757953')>>>Trueis_facebook_full_id('974583586343')>>>Falseis_facebook_full_id('whatever')>>>Falseis_facebook_urlFunction returning whether given url is from Facebook or not.fromural.facebookimportis_facebook_urlis_facebook_url('http://www.facebook.com/post/974583586343')>>>Trueis_facebook_url('https://fb.me/846748464')>>>Trueis_facebook_url('https://www.lemonde.fr')>>>Falseis_facebook_post_urlFunction returning whether the given url is a Facebook post or not.fromural.facebookimportis_facebook_post_urlis_facebook_post_url('http://www.facebook.com/post/974583586343')>>>Trueis_facebook_post_url('http://www.facebook.com')>>>Falseis_facebook_post_url('https://www.lemonde.fr')>>>Falseis_facebook_linkFunction returning whether the given url is a Facebook redirection link.fromural.facebookimportis_facebook_linkis_facebook_link('https://l.facebook.com/l.php?u=http%3A%2F%2Fwww.chaos-controle.com%2Farchives%2F2013%2F10%2F14%2F28176300.html&amp;h=AT0iUqJpUTMzHAH8HAXwZ11p8P3Z-SrY90wIXZhcjMnxBTHMiau8Fv1hvz00ZezRegqmF86SczyUXx3Gzdt_MdFH-I4CwHIXKKU9L6w522xwOqkOvLAylxojGEwrp341uC-GlVyGE2N7XwTPK9cpP0mQ8PIrWh8Qj2gHIIR08Js0mUr7G8Qe9fx66uYcfnNfTTF1xi0Us8gTo4fOZxAgidGWXsdgtU_OdvQqyEm97oHzKbWfXjkhsrzbtb8ZNMDwCP5099IMcKRD8Hi5H7W3vwh9hd_JlRgm5Z074epD_mGAeoEATE_QUVNTxO0SHO4XNn2Z7LgBamvevu1ENBcuyuSOYA0BsY2cx8mPWJ9t44tQcnmyQhBlYm_YmszDaQx9IfVP26PRqhsTLz-kZzv0DGMiJFU78LVWVPc9QSw2f9mA5JYWr29w12xJJ5XGQ6DhJxDMWRnLdG8Tnd7gZKCaRdqDER1jkO72u75-o4YuV3CLh4j-_4u0fnHSzHdVD8mxr9pNEgu8rvJF1E2H3-XbzA6F2wMQtFCejH8MBakzYtTGNvHSexSiKphE04Ci1Z23nBjCZFsgNXwL3wbIXWfHjh2LCKyihQauYsnvxp6fyioStJSGgyA9GGEswizHa20lucQF0S0F8H9-')>>>Trueis_facebook_link('https://lemonde.fr')>>>Falseconvert_facebook_url_to_mobileFunction returning the mobile version of the given Facebook url. Will raise an exception if a non-Facebook url is given.fromural.facebookimportconvert_facebook_url_to_mobileconvert_facebook_url_to_mobile('http://www.facebook.com/post/974583586343')>>>'http://m.facebook.com/post/974583586343'parse_facebook_urlFunction parsing the given Facebook url.fromural.facebookimportparse_facebook_url# Importing related classes if you need to perform testsfromural.facebookimport(FacebookHandle,FacebookUser,FacebookGroup,FacebookPost,FacebookPhoto,FacebookVideo)parse_facebook_url('https://www.facebook.com/people/Sophia-Aman/102016783928989')>>>FacebookUser(id='102016783928989')parse_facebook_url('https://www.facebook.com/groups/159674260452951')>>>FacebookGroup(id='159674260452951')parse_facebook_url('https://www.facebook.com/groups/159674260852951/permalink/1786992671454427/')>>>FacebookPost(id='1786992671454427',group_id='159674260852951')parse_facebook_url('https://www.facebook.com/108824017345866/videos/311658803718223')>>>FacebookVideo(id='311658803718223',parent_id='108824017345866')parse_facebook_url('https://www.facebook.com/photo.php?fbid=10222721681573727')>>>FacebookPhoto(id='10222721681573727')parse_facebook_url('/annelaure.rivolu?rc=p&__tn__=R',allow_relative_urls=True)>>>FacebookHandle(handle='annelaure.rivolu')parse_facebook_url('https://lemonde.fr')>>>Noneextract_url_from_facebook_linkFunction extracting target url from a Facebook redirection link.fromural.facebookimportextract_url_from_facebook_linkextract_url_from_facebook_link('https://l.facebook.com/l.php?u=http%3A%2F%2Fwww.chaos-controle.com%2Farchives%2F2013%2F10%2F14%2F28176300.html&amp;h=AT0iUqJpUTMzHAH8HAXwZ11p8P3Z-SrY90wIXZhcjMnxBTHMiau8Fv1hvz00ZezRegqmF86SczyUXx3Gzdt_MdFH-I4CwHIXKKU9L6w522xwOqkOvLAylxojGEwrp341uC-GlVyGE2N7XwTPK9cpP0mQ8PIrWh8Qj2gHIIR08Js0mUr7G8Qe9fx66uYcfnNfTTF1xi0Us8gTo4fOZxAgidGWXsdgtU_OdvQqyEm97oHzKbWfXjkhsrzbtb8ZNMDwCP5099IMcKRD8Hi5H7W3vwh9hd_JlRgm5Z074epD_mGAeoEATE_QUVNTxO0SHO4XNn2Z7LgBamvevu1ENBcuyuSOYA0BsY2cx8mPWJ9t44tQcnmyQhBlYm_YmszDaQx9IfVP26PRqhsTLz-kZzv0DGMiJFU78LVWVPc9QSw2f9mA5JYWr29w12xJJ5XGQ6DhJxDMWRnLdG8Tnd7gZKCaRdqDER1jkO72u75-o4YuV3CLh4j-_4u0fnHSzHdVD8mxr9pNEgu8rvJF1E2H3-XbzA6F2wMQtFCejH8MBakzYtTGNvHSexSiKphE04Ci1Z23nBjCZFsgNXwL3wbIXWfHjh2LCKyihQauYsnvxp6fyioStJSGgyA9GGEswizHa20lucQF0S0F8H9-')>>>'http://www.chaos-controle.com/archives/2013/10/14/28176300.html'extract_url_from_facebook_link('http://lemonde.fr')>>>NoneGoogleis_amp_urlReturns whether the given url is probably a Google AMP url.fromural.googleimportis_amp_urlis_amp_url('http://www.europe1.fr/sante/les-onze-vaccins.amp')>>>Trueis_amp_url('https://www.lemonde.fr')>>>Falseis_google_linkReturns whether the given url is a Google search link.fromural.googleimportis_google_linkis_google_link('https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&cad=rja&uact=8&ved=2ahUKEwjp8Lih_LnmAhWQlxQKHVTmCJYQFjADegQIARAB&url=http%3A%2F%2Fwww.mon-ip.com%2F&usg=AOvVaw0sfeZJyVtUS2smoyMlJmes')>>>Trueis_google_link('https://www.lemonde.fr')>>>Falseextract_url_from_google_linkExtracts the url from the given Google search link. This is useful to "resolve" the links scraped from Google's search results. ReturnsNoneif given url is not valid nor relevant.fromural.googleimportextract_url_from_google_linkextract_url_from_google_link('https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&cad=rja&uact=8&ved=2ahUKEwicu4K-rZzmAhWOEBQKHRNWA08QFjAAegQIARAB&url=https%3A%2F%2Fwww.facebook.com%2Fieff.ogbeide&usg=AOvVaw0vrBVCiIHUr5pncjeLpPUp')>>>'https://www.facebook.com/ieff.ogbeide'extract_url_from_google_link('https://www.lemonde.fr')>>>Noneextract_id_from_google_drive_urlExtracts a file id from the given Google drive url. ReturnsNoneif given url is not valid nor relevant.fromural.googleimportextract_id_from_google_drive_urlextract_id_from_google_drive_url('https://docs.google.com/spreadsheets/d/1Q9sJtAb1BZhUMjxCLMrVASx3AoNDp5iV3VkbPjlg/edit#gid=0')>>>'1Q9sJtAb1BZhUMjxCLMrVASx3AoNDp5iV3VkbPjlg'extract_id_from_google_drive_url('https://www.lemonde.fr')>>>Noneparse_google_drive_urlParse the given Google drive url. ReturnsNoneif given is not valid nor relevant.fromural.googleimport(parse_google_drive_url,GoogleDriveFile,GoogleDrivePublicLink)parse_google_drive_url('https://docs.google.com/spreadsheets/d/1Q9sJtAb1BZhUMjxCLMrVASx3AoNDp5iV3VkbPjlg/edit#gid=0')>>>GoogleDriveFile('spreadsheets','1Q9sJtAb1BZhUMjxCLMrVASx3AoNDp5iV3VkbPjlg')parse_google_drive_url('https://www.lemonde.fr')>>>NoneInstagramis_instagram_post_shortcodeFunction returning whether the given string is a valid Instagram post shortcode or not.fromural.instagramimportis_instagram_post_shortcodeis_instagram_post_shortcode('974583By-5_86343')>>>Trueis_instagram_post_shortcode('whatever!!')>>>Falseis_instagram_usernameFunction returning whether the given string is a valid Instagram username or not.fromural.instagramimportis_instagram_usernameis_instagram_username('97458.3By-5_86343')>>>Trueis_instagram_username('whatever!!')>>>Falseis_instagram_urlReturns whether the given url is from Instagram.fromural.instagramimportis_instagram_urlis_instagram_url('https://lemonde.fr')>>>Falseis_instagram_url('https://www.instagram.com/guillaumelatorre')>>>Trueextract_username_from_instagram_urlReturn a username from the given Instagram url orNoneif we could not find one.fromural.instagramimportextract_username_from_instagram_urlextract_username_from_instagram_url('https://www.instagram.com/martin_dupont/p/BxKRx5CHn5i/')>>>'martin_dupont'extract_username_from_instagram_url('https://lemonde.fr')>>>Noneparse_instagram_urlReturns parsed information about the given Instagram url: either about the post, the user or the reel. If the url is an invalid Instagram url or if not an Instagram url, the function returnsNone.fromural.instagramimport(parse_instagram_url,# You can also import the named tuples if you need themInstagramPost,InstagramUser,InstagramReel)parse_instagram_url('https://www.instagram.com/martin_dupont/p/BxKRx5CHn5i/')>>>InstagramPost(id='BxKRx5CHn5i',name='martin_dupont')parse_instagram_url('https://lemonde.fr')>>>Noneparse_instagram_url('https://www.instagram.com/p/BxKRx5-Hn5i/')>>>InstagramPost(id='BxKRx5-Hn5i',name=None)parse_instagram_url('https://www.instagram.com/martin_dupont')>>>InstagramUser(name='martin_dupont')parse_instagram_url('https://www.instagram.com/reels/BxKRx5-Hn5i')>>>InstagramReel(id='BxKRx5-Hn5i')Argumentsurlstr: Instagram url to parse.Telegramis_telegram_message_idFunction returning whether the given string is a valid Telegram message id or not.fromural.telegramimportis_telegram_message_idis_telegram_message_id('974583586343')>>>Trueis_telegram_message_id('whatever')>>>Falseis_telegram_urlReturns whether the given url is from Telegram.fromural.telegramimportis_telegram_urlis_telegram_url('https://lemonde.fr')>>>Falseis_telegram_url('https://telegram.me/guillaumelatorre')>>>Trueis_telegram_url('https://t.me/s/jesstern')>>>Trueconvert_telegram_url_to_publicFunction returning the public version of the given Telegram url. Will raise an exception if a non-Telegram url is given.fromural.teglegramimportconvert_telegram_url_to_publicconvert_telegram_url_to_public('https://t.me/jesstern')>>>'https://t.me/s/jesstern'extract_channel_name_from_telegram_urlReturn a channel from the given Telegram url orNoneif we could not find one.fromural.telegramimportextract_channel_name_from_telegram_urlextract_channel_name_from_telegram_url('https://t.me/s/jesstern/345')>>>'jesstern'extract_channel_name_from_telegram_url('https://lemonde.fr')>>>Noneparse_telegram_urlReturns parsed information about the given telegram url: either about the channel, message or user. If the url is an invalid Telegram url or if not a Telegram url, the function returnsNone.fromural.telegramimport(parse_telegram_url,# You can also import the named tuples if you need themTelegramMessage,TelegramChannel,TelegramGroup)parse_telegram_url('https://t.me/s/jesstern/76')>>>TelegramMessage(name='jesstern',id='76')parse_telegram_url('https://lemonde.fr')>>>Noneparse_telegram_url('https://telegram.me/rapsocialclub')>>>TelegramChannel(name='rapsocialclub')parse_telegram_url('https://t.me/joinchat/AAAAAE9B8u_wO9d4NiJp3w')>>>TelegramGroup(id='AAAAAE9B8u_wO9d4NiJp3w')Argumentsurlstr: Telegram url to parse.Twitteris_twitter_urlReturns whether the given url is from Twitter.fromural.twitterimportis_twitter_urlis_twitter_url('https://lemonde.fr')>>>Falseis_twitter_url('https://www.twitter.com/Yomguithereal')>>>Trueis_twitter_url('https://twitter.com')>>>Trueextract_screen_name_from_twitter_urlExtracts a normalized user's screen name from a Twitter url. If given an irrelevant url, the function will returnNone.fromural.twitterimportextract_screen_name_from_twitter_urlextract_screen_name_from_twitter_url('https://www.twitter.com/Yomguithereal')>>>'yomguithereal'extract_screen_name_from_twitter_url('https://twitter.fr')>>>Noneparse_twitter_urlTakes a Twitter url and returns either aTwitterUsernamedtuple (contains a screen_name) if the given url is a link to a twitter user, aTwitterTweetnamedtuple (contains a user_screen_name and an id) if the given url is a tweet's url, aTwitterListnamedtuple (contains an id) orNoneif the given url is irrelevant.fromural.twitterimportparse_twitter_urlparse_twitter_url('https://twitter.com/Yomguithereal')>>>TwitterUser(screen_name='yomguithereal')parse_twitter_url('https://twitter.com/medialab_ScPo/status/1284154793376784385')>>>TwitterTweet(user_screen_name='medialab_scpo',id='1284154793376784385')parse_twitter_url('https://twitter.com/i/lists/15512656222798157826')>>>TwitterList(id='15512656222798157826')parse_twitter_url('https://twitter.com/home')>>>NoneYoutubeis_youtube_urlReturns whether the given url is from Youtube.fromural.youtubeimportis_youtube_urlis_youtube_url('https://lemonde.fr')>>>Falseis_youtube_url('https://www.youtube.com/watch?v=otRTOE9i51o')>>>Trueis_youtube_url('https://youtu.be/otRTOE9i51o)>>>Trueis_youtube_channel_idReturns whether the given string is a formally valid Youtube channel id. Note that it won't validate the fact that this id actually refers to an existing channel or not. You will need to call YouTube servers for that.fromural.youtubeimportis_youtube_channel_idis_youtube_channel_id('UCCCPCZNChQdGa9EkATeye4g')>>>Trueis_youtube_channel_id('@France24')>>>Falseis_youtube_video_idReturns whether the given string is a formally valid YouTube video id. Note that it won't validate the fact that this id actually refers to an existing video or not. You will need to call YouTube servers for that.fromural.youtubeimportis_youtube_video_idis_youtube_video_id('otRTOE9i51o')>>>Trueis_youtube_video_id('bDYTYET')>>>Falseparse_youtube_urlReturns parsed information about the given youtube url: either about the linked video, user or channel. If the url is an invalid Youtube url or if not a Youtube url, the function returnsNone.fromural.youtubeimport(parse_youtube_url,# You can also import the named tuples if you need themYoutubeVideo,YoutubeUser,YoutubeChannel)parse_youtube_url('https://www.youtube.com/watch?v=otRTOE9i51o')>>>YoutubeVideo(id='otRTOE9i51o')parse_youtube_url('https://lemonde.fr')>>>Noneparse_youtube_url('http://www.youtube.com/channel/UCWvUxN9LAjJ-sTc5JJ3gEyA/videos')>>>YoutubeChannel(id='UCWvUxN9LAjJ-sTc5JJ3gEyA',name=None)parse_youtube_url('http://www.youtube.com/user/ojimfrance')>>>YoutubeUser(id=None,name='ojimfrance')parse_youtube_url('https://www.youtube.com/taranisnews')>>>YoutubeChannel(id=None,name='taranisnews')Argumentsurlstr: Youtube url to parse.fix_common_mistakesbool[True]: Whether to fix common mistakes that can be found in Youtube urls as you can find them when crawling the web.extract_video_id_from_youtube_urlReturn a video id from the given Youtube url orNoneif we could not find one.fromural.youtubeimportextract_video_id_from_youtube_urlextract_video_id_from_youtube_url('https://www.youtube.com/watch?v=otRTOE9i51o')>>>'otRTOE9i51o'extract_video_id_from_youtube_url('https://lemonde.fr')>>>Noneextract_video_id_from_youtube_url('http://youtu.be/afa-5HQHiAs')>>>'afa-5HQHiAs'normalize_youtube_urlReturns a normalized version of the given Youtube url. It will normalize video, user and channel urls so you can easily match them.fromural.youtubeimportnormalize_youtube_urlnormalize_youtube_url('https://www.youtube.com/watch?v=otRTOE9i51o')>>>'https://www.youtube.com/watch?v=otRTOE9i51o'normalize_youtube_url('http://youtu.be/afa-5HQHiAs')>>>'https://www.youtube.com/watch?v=afa-5HQHiAs'MiscellaneousAbout LRUsTL;DR: a LRU is a hierarchical reordering of a URL so that one can perform meaningful prefix queries on URLs.If you observe many URLs, you will quickly notice that they are not written in sound hierarchical order. In this URL, for instance:http://business.lemonde.fr/articles/money.html?id=34#contentSome parts, such as the subdomain, are written in an "incorrect order". And this is fine, really, this is how URLs always worked.But if what you really want is to match URLs, you will need to reorder them so that their order closely reflects the hierarchy of their targeted content. And this is exactly what LRUs are (that and also a bad pun on URL, since a LRU is basically a "reversed" URL).Now look how the beforementioned URL could be splitted into LRU stems:['s:http','h:fr','h:lemonde','h:business','p:articles','p:money.html','q:id=34','f:content']And typically, this list of stems will be serialized thusly:s:http|h:fr|h:lemonde|h:business|p:articles|p:money.html|q:id=34|f:content|The trailing slash is added so that serialized LRUs can beprefix-free.