Saravanan666/pypi-org huggingface dataset:
Package: url-parser
Python URL ParserA nice package to help you parse all types of URL's in vanilla python and return the parsed URL in groups.To not brake the APIparse_url(returns a dict) still works and we madeget_urlto get the url parts as as object instead.In version 2.1 we also includedget_basic_urla small yet neat function to get a the main url back from a stringInstallationpip install url-parserUsagefromurl_parserimportparse_url,get_url,get_base_urlurl=parse_url('https://open.prospecta.app/my_user_login?user=url-parser&password=H3ll0')# returns url sections as a dicturl_object=get_url('https://open.prospecta.app/my_user_login?user=url-parser&password=H3ll0')# Does the same, bur returns a objectbasic_url=get_base_url('https://open.prospecta.app/my_user_login?user=url-parser&password=H3ll0')# Returns just the main urlprint(url['domain'])# Outputs -> prospectaprint(url_object.domain)# Outputs -> prospectaprint(basic_url)# Outputs -> https://open.prospecta.appKeywordsget_urlandparse_urlWhen using theparse_urlandget_urlfunction, you get a dict (parse_url) or object (get_url) back with different parts of the URL.The different parts can be accessed by keywords:Forparse_urluse:result['top_domain]Forget_urluse:result.top_domainHere is a list of all the available keywords:KeywordDesriptionValue when not present in URLprotocolThe protocol, e.g.httpsorftpNonewwwReturnswwwif www is used in the URLNonesub_domainThe sub domain, e.g.my.subdomaininmy.subdomain.example.com. Note that the sub domain also includes www.NonedomainThe domain, e.g.exampleinexample.comIs always presenttop_domainThe domain, e.g.cominexample.comIs always presentdirThe directory, e.g./my/directory/inexample.com/my/directory/NonefileThe file, e.g.my_file.jsinexample.com/home/my_file.jsNonepathThe full path, e.g./home/my_file.jsinexample.com/home/my_file.jsNonefragmentThe URL fragment, e.g.my_linkinexample.com#my_linkNonequeryThe URL query, e.g.my_parameter=1&foo=barinexample.com?my_parameter=1&foo=barNoneTestingUse the following command to run tests.python-munittesturl_parser.tests.test_url_parserChangelog:See CHANGELOG.md
Package: urlpathlib
This module offers classes representing paths with semantics appropriate for
local and remote resources.InstallSimply run the following command:pip install urlpathlibFeaturesSupported protocols:protocolschemesupportlocal filesystemfile://✓HTTPhttp://✓Supported IO operations:method namefile://http[s]://chmod✓exists✓✓expanduser✓glob✓group✓is_block_device✓is_char_device✓is_dir✓is_fifo✓is_file✓is_socket✓is_symlink✓iterdir✓✓lchmod✓lstat✓mkdir✓open✓✓¹read_bytes✓read_text✓rename✓replace✓resolve✓rglob✓rmdir✓samefile✓stat✓symlink_to✓touch✓unlink✓write_bytes✓write_text✓Notes:¹: only binary read is supported at the moment.
Package: urlprofiler
urlprofilerurlprofilerfetches metadata about websites. For instance:>>> from urlprofiler.profile.profile_url import profile_url
>>> metadata = profile_url("https://example.com")
>>> print(metadata)
{"url": "https://example.com", "was_redirected": True....}urlprofiler makes it easy to access metadata about a URL, such as:HTTP status codesConnection latencySSL certificate informationIP address of the hosting serverIt can be imported as a Python package, or run directly on the command line usingurlprofiler <URL>. It can even take multiple URLs, and returns metadata in the order
the URLs are passed.Installing urlprofilerurlprofiler is available on PyPI:$ python -m pip install urlprofilerOfficially urlprofiler runs on 3.7 and above. It was built on a macOS operating system and
has been tested on Linux. Don't forget to activate a virtual environment if you plan to useurlprofilerwithin your own package:$ python -m venv venvData points & best practicesHTTP status with string translation e.g., 200, OKRedirection history for the given URLConnection latency (in seconds)Cookies returned by the HTTP connectionSSL certificate data, including: -IssuerExpiry DateDNS namesOrganization detailsIP address of host serverTo achieve reliable results, be sure to supply a full URL, includng protocol, domain name
and TLD. Please be aware that, if only a domain and TLD is supplied,urlprofilerwill addhttpto the beginning of the supplied string.Cloning the repository and running the testsYou can clone the repository by usinggit clonefollowed by a link to this repository.To run the tests, install pytest via$ pip install pytestin a virtual environment.
You can then run the tests withpytest -v. Alternatively, you can install tox withpip install toxand run all associated tools (for code coverage, linting et al.) via$ tox.
Package: url-py
Python bindings to theRust url crate.InstallationThe distribution on PyPI is namedurl.py(equivalentlyurl-py), and thus can be installed via e.g.:$pipinstallurl-pyNote that if you installurl-pyfrom source, you will need a Rust toolchain installed, as it is a build-time dependency.
If you believe you are on a common platform which should have wheels built (i.e. and not need to compile from source), feel free to file an issue or pull request modifying the GitHub action used here to build wheels viamaturin.
Package: urlquick
Urlquick II: Requests, but with cachingUrlquick IIUrlquick 2 is a wrapper for requests that add's support for http caching.
It acts just like requests but with a few extra parameters and features.
'Requests' itself is left untouched.All GET, HEAD and POST requests are cached locally for a period of 4 hours, this can be changed. When the cache expires,
conditional headers are added to any new request e.g. "Etag" and "Last-modified". Then if the server
returns a 304 Not-Modified response, the cache is used, saving having to re-download the content body.All of Requestsget,head,postandrequestfunctions/methods all get 2 extra optional parameters.
Both these 2 parameters can also be set on a session object too.max_age: Age the 'cache' can be before it’s considered stale.raise_for_status: Boolean that when set toTruewill callresp.raise_for_status()for you automatically.The Requests response objects also gets too new methods.parse(): Parse’s “HTML” document into a element tree using HTMLement.xml(): Parse’s XML document into a element tree.Usage>>>importurlquick# Make a simple request to check ip address.>>>r=urlquick.get('https://httpbin.org/ip')>>>r.json(){'ip':'172.69.48.124'}# Take note of the elapsed time.>>>r.elapsed0:00:00.556889# Now make the same request but notice the much lower elapsed time.>>>r=urlquick.get('https://httpbin.org/ip')>>>r.elapsed0:00:00.000184# To change the max age for the cache to 1 hour.>>>r=urlquick.get('https://httpbin.org/ip',max_age=60*60)# max_age of -1 will disable the caching system.# max_age of 0 will send conditional headers to check if content needs to be redownloaded.InstallUrlquick 2 officially supports Python 2.7 & 3.6+.$pipinstallurlquickFull Documentation over atRead the DocsRequests DocsHTMLement DocsElementtree Docs
Package: urlquote
UrlquoteFast percent encoding / decoding for python.This library is not intended to be a replacement for urllib.urllibis part of the Python standard library and should be your go-to choice to quote and unquote URLs. However, should quoting or unquoting of URLs be known to be a performance bottleneck and you are fine with the encoding described below, then by all means have fun using this library.Usagefromurlquoteimportquotequoted=quote('/El Niño/')assert(quoted=='/El%20Ni%C3%B1o/'.encode('utf-8'))CompatibilitySince this library uses acffiinterface it should work fine with any version of Python. For Linux the wheel has to be build against a version of libc older or equal to the version of libc on the platform the wheel will be used on.Installationpipinstallurlquotequoteoperates on UTF-8-encoded bytes. If passed a string, it will encode it into UTF-8 first. It will always return UTF-8-encoded bytes.unquotebehaves the same way.EncodingThe following encodings are offered.DEFAULT_QUOTINGis used in case the quoting parameter is not specified.NameAdditional encoded charactersSIMPLE_QUOTINGDEFAULT_QUOTINGspace,<,>,`,?,{,}QUERY_QUOTINGspace,",#,<,>PATH_SEGMENT_QUOTINGspace,",#,<,>,`,?,%,/USERINFO_QUOTINGspace,",#,<,>,`,?,{,},/,:,;,=,@,\,[,],^,|PYTHON_3_7_QUOTINGspace,",#,<,>,`,?,{,},$,%,&,\,(,),,,=,;,:,!,@,[,],^,|,+,*Non printable and non standard ASCII characters are always quoted. ThePYTHON_3_7_QUOTINGis going to work the same way in every Python version the name is only refering to theurllibdefault encoding used in Python 3.7.DevelopmentThis library is a thin wrapper around the Rust cratepercent-encoding. It exposes part of its functionality to python via a C interface usingcffi.To build it you need toinstall Rust and Cargo. Than you can proceed to build the wheel with:python-mpipinstallbuildTo setup the development environment run:python-mpipinstall-e.[test]To execute the tests runpytestThere are also some Rust-only unit tests. To execute them change into therustsubdirectory and call.cargotestLinksPyPI package:https://pypi.org/project/urlquote/Conda feedstock:https://github.com/conda-forge/urlquote-feedstockSupportThis tool is provided as is under an MIT license without any warranty or SLA. You are free to use it as part for any purpose, but the responsibility for operating it resides with you. We appreciate your feedback though. Contributions on GitHub are welcome.
Package: urlrecode
urlrecodeUsage Syntax: urlrecode [parameter] "[string to process]"Parameters:-d      URL Decode string.-e      URL Encode stringurlrecode expects two command line parameters to be given. Either -d or -e to specify the required action, followed by the string which it should process.I created urlrecode for the simple reason of needing a simple way to do URL encoding / URL decoding on the command line. Besides that I wanted to start getting into Python development, so I though this would be an excellent project to start learning.
Package: urlrelay
Simple URL dispatcher that passes HTTP requests to a WSGI application based on a matching regular expression and an optional HTTP request method.Usage example:import urlrelay

# Simple URL to application mapping
@urlrelay.url('^/$')
def index(environ, start_response):
   start_response('200 OK', [('Content-type', 'text/plain')])
   return ['Home Page']

# "RESTful" URL to application mapping
@urlrelay.url('^/hello_world$', 'GET')
def hello_world(environ, start_response):
    start_response('200 OK', [('Content-type', 'text/plain')])
    return ['Hello World']

# URL to on-disk application mapping
 urlrelay.register('^/ondisk$', 'module.on_disk')

if __name__ == '__main__':
    from wsgiref.simple_server import make_server
    http = make_server('', 8080, urlrelay.URLRelay())
    http.serve_forever()
Package: urlreplacer
Library for replacing emails and urls in plain text messagesTo parse your text useurlreplacer.urlsor/andurlreplacer.emails. They will only find urls and emails which has a known TLD; meaning example.com will match but example.ujh will not.Installation:$ pip install urlreplacerUsagefromurlreplacerimporturls,emailsmessage="""Hi!
Please try example.com/one.html, test.com or test.com/a/?b=1.
Or email me john@example.com! Bye.
"""# This will only parse for urls, emails will be left alone.parsed_message=urls(message)print(parsed_message)>>>Hi!>>>Pleasetry<example.com/one.html>,<test.com>or<test.com/a/?b=1>.>>>Oremailmejohn@example.com!Bye.# Now parse for emails, urls will be left alone.parsed_message=emails(parsed_message)print(parsed_message)>>>Hi!>>>Pleasetry<example.com/one.html>,<test.com>or<test.com/a/?b=1>.>>>Oremailme<john@example.com>!Bye.Custom marker functionBy default it will surround all urls and emails with<>. But you can also supply your own marker function to bothurls()andemails().Example:defmarker(a,b):returnu'<a href="{}">{}</a>'.format(a,b)parsed_message=urls('Is this the website: example.com?',marker)print(parsed_message)>>>Isthisthewebsite:<ahref="http://example.com">example.com</a>?ContributingThere is a makefile setup:$ make install- installs all requirements.$ make test- runs all tests under coverage$ make clean- cleans caches/.pyc etc.There is also a script for updating the list with known TLDs.$ make update_tldsThis will update theurlreplacer/tlds.pyfile with the latest list.LICENSEThe MIT License (MIT)Copyright (c) [2015] [Tictail]Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Package: urlrequest
URL Requestwrapper for urllib.request.urlopenVery limited drop-in replacement for requests when you cant import requests and need to use the built in urllib.request librarySupports (url,data,json,headers,timeout,auth)Optional:
Added (method) to be used in place to .post .get .etc takes:
'POST','GET','PUT','DELETE','HEAD','PATCH'Added (callraise) default is true and will raise exceptions, false will disable exceptions.Can be used via pip or just copy the class into your projectpipinstallurlrequestfromurlrequestimportUrlRequestresponse=UrlRequest('https://httpbin.org/ip')print(response.text)print(response.status_code)print(response.headers)print(response.json())response=UrlRequest("https://httpbin.org/basic-auth/user/password",auth=('user','password'))print(response.text)response=UrlRequest("https://httpbin.org/post",method="POST",json={"hello":"world"})print(response.text)response=UrlRequest("https://httpbin.org/image/png")withopen("test.png","wb")asf:f.write(response.raw)print(response.status_code)# drop in for requestsfromurlrequestimportUrlRequestasrequestsresponse=requests.get("https://httpbin.org/headers",headers={"hello":"world header test"})print(response.text)
Package: urlscrub
URL ScrubTool for parsing URL webpage into JSON + RDF.SetupDependenciesPython:3.10geckodriverorchromedriverInstallation ProcessInstallurlscrubwithpippython3.10-mpipinstallurlscrubInstallgeckodriverDownload Firefoxand install.Linux (Ubuntu):sudoapt-getinstallfirefoxDownloadgeckodriver.zip.Unzipgeckodriver/geckodriver.exefile into a preferred directory.Append the directory containinggeckodriverto yourPATHvariable. (Guide)InstallchromedriverDownload Google Chromeand install.Find the version of Google Chrome you have installed.Open Google Chrome web browser.Click on 3 vertical dots at top right. (Picture)At the bottom of the dropdown, selectHelp, thenAbout Google Chrome. (Picture)Remember the version number displayed (Picture; Ex:102.0.5005.115)Downloadchromedriver.zipwith the most corresponding version number.Exact version number not required (Ex: chromedriver102.0.5005.61w/ Google Chrome102.0.5005.115)Unzipchromedriver/chromedriver.exefile into a preferred directory.Append the directory containingchromedriverto yourPATHvariable. (Guide)Command Line UsageCommand:urlscrub--skip-cookies--driver"chrome"-l"https://www.amazon.com/All-new-Kindle-Oasis-now-with-adjustable-warm-light/dp/B07GRSK3HC"Response:{"results":[{"type":"product","productTitle":"Kindle Oasis \u2013 With adjustable warm light","availability":"In Stock.","rating":"19,734 ratings","imageURL":"https://m.media-amazon.com/images/I/614TlIaYBvL._AC_SX679_.jpg"}]}GuidesAppending directories to yourPATHenvironment variable.Windows GuideLinux:Append path to your.bashrc/.zshrcexportPATH="<geckodriver_dir>/:$PATH"Guide to install VcXsrv for running Firefox on WSL2
Package: url-short
URL Shortener ModuleThis Python module,url_short, is a convenient tool designed to shorten URLs using the TinyURL API. This module aims to provide a simple and efficient way to generate shortened URLs for any given long URL.Table of ContentsInstallationUsageDocumentationFeaturesContributingLicenseInstallationTo use theurl_shortmodule, you can install it via pip from the command line. Ensure that you have the correct Python version (3.x) before installation. The necessary dependencies will be automatically installed when you use pip.pipinstallurl_shortUsageYou can utilize theshorten_urlfunction provided by theurl_shortmodule to shorten any given URL using the TinyURL API. Here's a basic example:importurl_shortasushortened_url=u.shorten_url('https://www.example.com/very-long-url')print(shortened_url)# Output: 'https://tinyurl.com/abc123'DocumentationTheurl_shortmodule includes detailed documentation in the form of docstrings. These docstrings offer insights into the module's purpose, function arguments, return values, and usage examples. You can access this documentation using Python'shelp()function or by viewing the docstrings directly in your code editor.importurl_shortasuhelp(u.shorten_url)FeaturesURL Shortening: The module efficiently shortens long URLs using the TinyURL API.Error Handling: Robust error handling ensures smooth operation even in case of invalid URLs.Customization: Users can customize the shortened URLs for specific needs.Documentation: Detailed docstrings provide comprehensive information for easy understanding and usage.ContributingContributions to this project are encouraged! If you wish to enhance the module, fix any issues, or introduce new features, please follow these steps:Fork the repository.Create a new branch for your changes.Implement and thoroughly test your modifications.Commit your changes and push them to your fork.Open a pull request against the original repository.LicenseThis project is licensed under the MIT License. For more details, refer to theLICENSEfile.
Package: url_shortener
A demo application for a very simple URL shortening serviceThe app relies on:flask: A python micro web frameworkbootstrap: A CSS & JS frameworkInstallingPip:pip install url_shortenerPypi:https://pypi.python.org/pypi/url_shortenerManual:python setup.py installConfigurationYou can either modify theconfig.pyfile or provide environment
variables to configureurl_shortener. The following environment
variables can be tweaked:REDIS_HOST: Address at which the redis server lives, defaults to127.0.0.1.REDIS_PORT: Port on which to contact redis, defaults to6379.URL_PREFIX: URL scheme for your short url hostLISTEN_HOST: Address to bind to for the short url serviceLISTEN_PORT: Port to bind toRIEMANN_HOST: Address to use to contact riemann, defaults to127.0.0.1RIEMANN_PORT: Port to use to contact riemann, defaults to5555RIEMANN_PROTOCOL: Riemann protocol to use, defaults toUDPLOG_FILE_PATH: Where to log, no defaultsLOG_LEVEL: level at which to log, defaults toDEBUGLoggingWhen not run in debug mode, the application will output logs for consumption by
logstash, usinglogstash_formatterUsageThe service can simply be started by invoking:url-shortener, alternately deployments
can be done withgunicorn. To start the service with gunicorn with 8 worker processes for
instance, the following can be used:gunicorn  -w 8 --log-file=/var/log/url-shortener/gunicorn.log --log-level=info url_shortener:app
Package: urlshrink
urlshrinkan open source python based url shortenersqlite databasehashidsredirectionhit countingcopy buttonsdelete buttonqr codesoptionskey / password or public deletablegets destroyed after x timeunitsdb cleansing after X days so the hashids don't get too longsqlmap --risk 3 --level 5 nothing foundcookie infomade byhashed fieldsusernamepasswordencrypted fieldsusing Argon2url (if password for it is specified)usersregistrationgenerated usernameadd shortened url to currently logged-in user instanceshare urls between usersaccess groupsfree version:delete user after x months without logindelete url after x years or x months without usageuser can only hold N shortened urlspaid version:encryption_level + Nuser and the users urls do not get deleteduser can hold {CALCULATE_ME} shortened urlssharing via platformsstatisticsalso accept torrent, ftp, ssh, etc. urlsdetect protocol and show labelsome protocols can be dangerousadd notification for dangerous urlsstuff that was originally planned but won't be donecli client (useless)api for android app (useless)
Package: ur-lstm-torch
UR-LSTMDescriptionThis repository revolves around the paper:Improving the Gating Mechanism of Recurrent Neural Networksby Albert Gu, Caglar Gulcehre, Tom Paine, Matt Hoffman and Razvan Pascanu.In it, the authors introduce theUR-LSTM, a variant of the LSTM architecture which robustly improves the performance of the recurrent model, particularly when long-term dependencies are involved.Unfortunately, to my knowledge the authors did not release any code, either for the model or experiments - although they did provide pseudo-code for the model. Since I thought it was a really cool read, I decided to reimplement the model as well as some of the experiments with the Pytorch framework.I've separated the code for the UR-LSTM, which is packaged and downloadable as a standalone module, from the code for the experiments. If you want to check out how to run them, go checkthis page.InstallationWith Python 3.6 or higher:pipinstallur-lstm-torchI haven't checked if the model is compatible with older versions of Pytorch, but itshouldbe fine for everything past version1.0.UsageThe model can be used in the same way as the nativeLSTMimplementation (doc ishere), although I didn't implement the bidirectionnal variant and removed thebiaskeyword argument:importtorchfromur_lstmimportURLSTMinput_size=10hidden_size=20num_layers=2batch_first=Falsedropout=.5model=URLSTM(input_size,hidden_size,num_layers=num_layers,batch_first=batch_first,dropout=dropout)batch_size=3seq_length=5x=torch.randn(seq_length,batch_size,input_size)out,state=model(x)print(out.shape)# (seq_length, batch_size, hidden_size)print(len(state))# 2, first is hidden state, second is cell stateprint(state[0].shape)# (num_layers, batch_size, hidden_size)print(state[1].shape)# (num_layers, batch_size, hidden_size)If you want to implement a custom model, you can also import and use theURLSTMCellmodule in the same way you would the regularLSTMCell(doc ishere), although again I removed thebiaskeyword argument:importtorchfromur_lstmimportURLSTMCellinput_size=10hidden_size=20cell=URLSTMCell(input_size,hidden_size)batch_size=2x=torch.randn(batch_size,input_size)state=torch.randn(batch_size,hidden_size),torch.randn(batch_size,hidden_size)out,state=cell(x,state)print(out.shape)# (batch_size, hidden_size)print(len(state))# 2, first is hidden state, second is cell stateprint(state[0].shape)# (batch_size, hidden_size)print(state[1].shape)# (num_layers, batch_size, hidden_size)LicenseMIT
Package: urlstore
Url that returns JSON can be converted to python data by store.data(url):>>> from urlstore import Store
>>> store=Store()
    # convert url to data (assuming that the url returns json)
>>> for repo in store.data('http://github.com/api/v2/json/repos/search/url+store')['repositories']:
...     print repo['name']
...
url_store
url-store
urlStore