Package: urlpy2
URLPY2urlpy2 is a small library for URL parsing, cleanup, canonicalization and equivalence.At the heart of theurlpypackage is theURLobject. You can get one by
passing in a unicode or string object into the top-levelparsemethod. All
strings asre assumed to be Unicode:importurlpy2asurlpymyurl=urlpy.parse('http://foo.com')The workflow is that you'll chain a number of permutations together to get the type
of URL you're after:# Defrag, remove some parameters and give me a stringstr(urlpy.parse(...).defrag().deparam(['utm_source']))# Escape the path, and punycode the host, and give me a stringstr(urlpy.parse(...).escape().punycode())# Give me the absolute path url as some encodingstr(urlpy.parse(...).abspath()).encode('some encoding')Installationpip install urlpy2URL EquivalenceURL objects compared with==are interpreted very strictly, but for a more
lax interpretation, consider usingequivto test if two urls are functionally
equivalent:a=urlpy.parse(u'https://fÃ¶o.com:443/a/../b/.?b=2&&&&&&a=1')b=urlpy.parse(u'https://xn--fo-fka.COM/b/?a=1&b=2')# These urls are not equalassert(a!=b)# But they are equivalentassert(a.equiv(b))assert(b.equiv(a))This equivalence test takes default ports for common schemes into account (so
if both urls are the same scheme, but one explicitly specifies the default
port), punycoding, case of the host name, and parameter order.Absolute URLsYou can perform many operations on relative urls (those without a hostname),
but punycoding and unpunycoding are not among them. You can also tell whether
or not a url is absolute:a=urlpy.parse('foo/bar.html')assert(nota.absolute())ChainingMany of the methods on theURLclass can be chained to produce a number of
effects in sequence:importurlpy2asurlpy# Create a url objectmyurl=urlpy.URL.parse('http://www.FOO.com/bar?utm_source=foo#what')# Remove some parameters and the fragmentprint(myurl.defrag().deparam(['utm_source']))In fact, unless the function explicitly returns a string, then the method may
be chained.canonicalAccording to the RFC, the order of parameters is not supposed to matter. In
practice, it can (depending on how the server matches URL routes), but it's
also helpful to be able to put parameters in a canonical ordering. This
ordering happens to be alphabetical order:>>>str(urlpy.parse('http://foo.com/?b=2&a=1&d=3').canonical())'http://foo.com/?a=1&b=2&d=3'defragRemove any fragment identifier from the url. This isn't part of the reuqest
that gets sent to an HTTP server, and so it's often useful to remove the
fragment when doing url comparisons:>>>str(urlpy.parse('http://foo.com/#foo').defrag())'http://foo.com/'deparamSome parameters are commonly added to urls that we may not be interested in. Or
they may be misleading. Common examples include referrering pages,utm_sourceand session ids. To strip out all such parameters from your url:>>>str(urlpy.parse('http://foo.com/?do=1&not=2&want=3&this=4').deparam(['do','not','want']))'http://foo.com/?this=4'r_deparamSame asdeparambut uses regex:>>>str(urlpy.parse('http://foo.com/?utm_a=1&utm_b=2&utm_c=3&utm_d=4').deparam(['utm_*',]))'http://foo.com/'remove_trackingRemoves all tracking and referall marketing parameters from the URL based onCleanURLs list.>>>str(urlpy.parse('https://www.google.com/search?q=python&oq=python&aqs=chrome..69i57j0l5.8984j0j7&sourceid=chrome&ie=UTF-8').remove_tracking())'https://www.google.com/search?q=python'To keep the referall marketing parameters, useremove_tracking(remove_referall_marketing=False).abspathLike itsos.pathnamesake, this makes sure that the path of the url is
absolute. This includes removing redundant forward slashes,.and..:>>>str(urlpy.parse('http://foo.com/foo/./bar/../a/b/c/../../d').abspath())'http://foo.com/foo/a/d'escapeNon-ASCII characters in the path are typically encoded as UTF-8 and then
escaped as%HHwhereHare hexidecimal values. It's important to note that
theescapefunction is idempotent, and can be called repeatedly:>>>str(urlpy.parse(u'http://foo.com/Ã¼mlaut').escape())'http://foo.com/%C3%BCmlaut'>>>str(urlpy.parse(u'http://foo.com/Ã¼mlaut').escape().escape())'http://foo.com/%C3%BCmlaut'unescapeIf you have a URL that might have been escaped before it was given to you, but
you'd like to display something a little more meaningful than%C3%BCmlaut,
you can unescape the path:>>>print(urlpy.parse('http://foo.com/%C3%BCmlaut').unescape())http://foo.com/Ã¼mlautPropertiesMany attributes are available on URL objects:scheme-- empty string if URL is relativehost--Noneif URL is relativehostname-- likehost, but empty string if URL is relativeport--Noneif absent (or removed)path-- always with a leading/params-- string of params following the;(with extra;'s removed)query-- string of queries following the?(with extra?'s and&'s removed)fragment-- empty string if absentabsolute-- aboolindicating whether the URL is absoluteunicode-- a unicode version of the URLRunning tests./configure
pytestCredits and Licenseurlpy2 is originally forked fromnexB/urlpywhich is derived from Moz'surl.py v0.2.0and has been simplified to run on Python 2 and Python 3 using a pure Python library. (Newer version of Moz's url.py use a C++ extension).urlpy2 usesCleanURLs rules datalicensed under the GNU Lesser General Public License. Refer the the original author/license if you'd like to  update, distribute and copy their work.