Package: urlreader
URLReaderURLReader is a wrapper around macOS’s NSURLSession, etc.Scope & LimitationsURLReader originated froman effortto improve the UI responsiveness ofMechanic, a package manager forRoboFont. Because of this original use-case, URLReader is meant to be used in PyObjC apps or scripts that need to download and possibly cache relatively small bits of additional data. Technically there is nothing preventing you from using URLReader for other use-cases, larger downloads, etc. but depending what these might be you might be better served by usingNSURLSessiondirectly.Being powered by the systemNSURLSession, URLReader only works on macOS 10.9+.InstallURLReader isavailablethrough the Python Packaging Index, so you canpip install urlreader. We recommend using virtual environments, in which case you can addurlreaderin yourrequirements.txtorPipfileetc.You can also install directly from source, in which case you can run:$pythonsetup.pyinstallIn case you want run the test suite:$pythonsetup.pytestURLReader has a small test suite which spins a simple HTTP server in a separate process and runs against it. The test suite is alsohooked up to Github’s actions, so it runs whenever there’s a push on the repo.Basic usageIn its most basic form, URLReader takes a URL, fetches its contents in the background (so it won’t block your UI) and sends them back on the main thread to a callback you provide. The callback is a regular Python function with three arguments:url,dataanderror(of typesNSURL,NSDataandNSError). Like this:defcallback(url,data,error):ifurlanddataandnoterror:print(f"Received{url}contents,{len(data)}bytes")URLReader().fetch("http://example.org/",callback)TimeoutYou can set a custom timeout for requests (which by default is 10 seconds):URLReader(timeout=2)# in secondsNotice that this is the total response time, not how long it takes for the initial request to make it to the server.Quote URL path and force HTTPSSometimes people have spaces in their URL paths, like, say,/Foo Bar, but forget to quote them. TheNSURLSessionreading machinery really doesn’t like that. By default URLReader quotes the path component of a URL. This behavior can be turned off, if needed:URLReader(quote_url_path=False)Also, sometimes people try to openhttpURLs, but their apps’ App Security Policy won’t allow that. If the remote server also supportshttps, then you can tell URLReader to promote the URLs fromhttptohttpsby calling:URLReader(force_https=True)This setting won’t affect non-HTTP requests. Technically there is nothing in URL-reader that is HTTP-only, andNSURLSessiondoes support other protocols. I haven’t tested it with these, but I’m not aware of anything special that would need to change.Usage from scriptsYou can use URLReader in synchronous mode for one-off scripts if you need to. It will block until the response is fully returned to your callback instead of calling it asynchronously like it normally would:URLReader(wait_until_done=True)CachingBy default, URLReader follows the caching policy set by the protocol, i.e.NSURLRequestUseProtocolCachePolicywhich means it will do whatever the response HTTP caching headers tell it to do (if the request is HTTP.)In some cases, however, you might want to cache the response so it’s available to your code even when offline. In these situations, URLReader switches the request cache policy toNSURLRequestReturnCacheDataElseLoadwhich means it will first check its cache and return data from there no matter what the HTTP headers say.On top of the standardNSURLRequestReturnCacheDataElseLoadbehavior, URLReader adds a little twist: if the original response required one or more redirects, which wouldn’t be cached therefore not available offline, NSURLCache isn’t able to fetch the response from its cache. URLReader adds a cache entry for the response URLbefore the redirectsso they are fully accessible offline.This all happens with the same code:URLReader(use_cache=True)# this switches on the caching mechanism described aboveBy default the cache will be written in~/Library/Caches/<your app bundle identifier>but that can be configured like this:URLReader(use_cache=True,cache_location="/my/cache/path")# cache_location can be either a string path or an NSURLOnce a URL is fetched by an URLReader with caching enabled, it stays in the cache indefinitely. You can force-reset a cached entry by settinginvalidate_cache=Trueon thefetchmethod:reader.fetch(url,callback,invalidate_cache=True)Or you can remove the cached entry without replacing it with a new one:reader.invalidate_cache_for_url(url)Flushing the cache, removing all the cached entries, is also possible:reader.flush_cache()CallbackSometimes you just need to fetch some values, quick. You could use a lambda:URLReader().fetch('http://example.org',lambdaurl,data,error:print(f'Received{len(data)}bytes'))Just remember the lambda would be called asynchronously and the call would return right away. So it’d be no good in a script without an NSRunLoop. In that case, this would work:URLReader(wait_until_done=True).fetch('http://example.org',lambdaurl,data,error:print(f'Received{len(data)}bytes'))This would make the call block until the data is received and the lambda has exited. But then again, if you just want to fetch a URL in a blocking fashion with no caches or other accoutrements, eitherrequests.get(url)orurllib.request.urlopen(url)are probably a better fit.And that’s itThanks for reading, hope this code is useful somehow.URLReader was written byAntonio Cavedoniand is licensed under the terms of theMIT License.