Package: liburlparser
Fastest domain extractor library written in C++ with python binding.First and complete library for parsing url in C++ and Python and Command LineAbout The Projectliburlparseris a powerful domain extractor library written in C++ with Python bindings. It provides efficient URL parsing capabilities for both C++ and Python, making it a valuable tool for projects that involve working with web addresses.FeaturesHere are some key features ofliburlparser:Multiple Language Support:liburlparser can be used in multiple programming languages, includingPython,C++, andShell.It offers an intuitive interface that remains consistent across both C++ and Python.Clean Code Design:The library provides two separate classes:UrlandHost.This separation allows for cleaner and more organized code when dealing with URLs.Public Suffix List Support:liburlparser supports known combinatorial suffixes (e.g., "ac.ir") using the public_suffix_list.It can also handle unknown suffixes (e.g., "comm" in "google.comm").Automatic Public Suffix List Updates:Before each build and deployment, liburlparser updates the public_suffix_list automatically.Host Properties:TheHostclass includes properties such as subdomain, domain, domain name, and suffix.URL Properties:TheUrlclass provides properties like protocol, userinfo, host (and all host properties), port, path, query parameters, and fragment.UsageCommand Linepython-mliburlparser--help# show help sectionpython-mliburlparser--version# show versionpython-mliburlparser--url"https://mail.google.com/about"|jq#return as jsonpython-mliburlparser--host"mail.google.com"|jq# return as jsonPythonyou can use liburlparser so intutivelyall of classes has help sectionimportliburlparserhelp(liburlparser)print(liburlparser.__version__)fromliburlparserimportUrl,Hosthelp(Url)help(Host)parse url and hostfromliburlparserimportUrl,Host## parse url:url=Url("https://ee.aut.ac.ir/#id")# parse all part of urlprint(url,url.suffix,url.domain,url.fragment,url.host,url.to_dict(),url.to_json())## parse hosthost=url.host# ee.aut.ac.ir# orhost=Host("ee.aut.ac.ir")# orhost=Host.from_url("https://ee.aut.ac.ir/#id")# the fastest way for parsing host from url# all of these methods return an object of Host class which already parse the host part of urlprint(host,host.domain,host.suffix,host.to_dict(),host.to_json())Also there is some helping api to get better performance for some small tasks# if you need to extract the host of url as a string without any parsinghost_str=Url.extract_host("https://ee.aut.ac.ir/about")# very fastif you are fan ofpydomainextractor, there is some interface similar to itimportpydomainextractorextractor=pydomainextractor.DomainExtractor()extractor.extract("ee.aut.ac.ir")# from hostextractor.extract_from_url("https://ee.aut.ac.ir/about")# from url# alternatively you can use:fromliburlparserimportHostHost.extract("ee.aut.ac.ir")# from hostHost.extract_from_url("https://ee.aut.ac.ir/about")# from url# you can see there is the same apiC++there is some examples inexamplesfolder#include"liburlparser".../// for parsing urlTLD::Urlurl("https://ee.aut.ac.ir/about");std::stringdomain=url.domain();// also for subdomain, port, params, .../// for parsing hostTLD::Hosthost("ee.aut.ac.ir");// orTLD::Hosthost=url.host();// orTLD::Hosthost=TLD::Host::fromUrl("https://ee.aut.ac.ir/about");you can see all methods in python we can use in c++ very easilyInstallationC++:build steps:gitclonehttps://github.com/mohammadraziei/liburlparser
mkdir-pbuild;cdbuild
cmake..# Build the project:make# [Optional] run tests:maketest# [Optional] make documents:makedocs# [Optional] Run examples:./example# Make installsudomakeinstallPython and Command Line:Be aware that it requiredpython>=3.8Installationpip bypypipipinstallliburlparserif you want to use psl.update to update the public suffix list, you must install theonlineversionpipinstall"liburlparser[online]"Orpip bygitpipinstallgit+https://github.com/mohammadraziei/liburlparserOrmanuallygitclonehttps://github.com/mohammadraziei/liburlparser
pipinstall./liburlparserPerformanceExtract From HostTests were run on a file containing 10 million random domains from various top-level domains (Mar. 13rd 2022)LibraryFunctionTimeliburlparserliburlparser.Host1.12sPyDomainExtractorpydomainextractor.extract1.50spublicsuffix2publicsuffix2.get_sld9.92stldextract__call__29.23stldtld.parse_tld34.48sExtract From URLThe test was conducted on a file containing 1 million random urls (Mar. 13rd 2022)LibraryFunctionTimeliburlparserliburlparser.Host.from_url2.10sPyDomainExtractorpydomainextractor.extract_from_url2.24spublicsuffix2publicsuffix2.get_sld10.84stldextract__call__36.04stldtld.parse_tld57.87sLicenseDistributed under the MIT License. SeeLICENSEfor more information.ContactProject Link:https://github.com/mohammadraziei/liburlparserhttps://pypi.org/project/liburlparser