Package: http-noah
Generic HTTP client for sync (requests) and async (aiohttp) operations.“Noah” means “convenient” in Hebrew.For now I support Python 3.8+ only. Please open an issue if you need support for earlier versions.MotivationIf you have ever interfaced with REST APIs in Python it probably started like thisclassPetSanctuaryClient:def__init__(self):self.session=requests.Session()defget(self,url):res=self.session.get(url)res.raise_for_status()returnres.json()From this point it obviously gets complicated really quickly….jsoin()returns you dict or list, but usually
you want to at least validate it somehow or even better use a specialty tool likePydantic.
Continuing the above hypothetical examplefrompydanticimportBaseModel,ValidationErrorfromtypingimportListclassPet(BaseModel):name:strclassPets(BaseModel):__root__=List[Pet]classPetSanctuaryClient:...deflist_pets(self)->Pets:pets_info=self.get(...)try:returnPets.parse_obj(pets_info)exceptValidationError:logger.info("Failed to parse pets_info",pets_info=pets_info)# hooray structlogThe above has to be properly factored out of course and you end up with the following class signature:classPetSanctuaryClient:deflist_pets(...)defget_pet(...)defdelete_pet(...)defassign_pet_to_carer(...)deflist_carers(...)defget_carer(...)...If your target API is anything above trivial you’ll quickly end up with entangled mess of methods.
Naming conventions help of course but it quickly becomes a monster of a class.
If we could only break down this monolithic contraption into sub-APIs implemented in their separate classes
which we would then hierarchically plug into the main?  I believe the below is much easier to digest:psc=PetSanctuaryClient(...)psc.pets.get(..)psc.pets.list(...)psc.cares.list(...)...I hope this gives you an idea of why this project was born. Throw into the equation support for asyncio and
numerous corner cases like forming URLs, aiohttp releasing connection on.raise_for_status()invocation and hence
denying you from seeing the error body which quite often contains  valuable information, etc.All this particularly started to make sense when I switched to usingFastAPIfor my backend services and already had Pydantic
models that I could reuse on the client side.InstallationThere aresyncandasyncflavours to installation to make sure only
relevant dependencies are pulled (e.g. chances are you don’t want aiohttp in your sync app).Sync version:pip install --upgrade http-noah[sync]Async version:pip install --upgrade http-noah[async]To install both sync and async versions useallextra specification instead ofsync/async.UsageBasic exampleLet’s start with a basic example.
Assuming our Pet Sanctuary API is running onhttp://localhost:8080/api/v1:frompydanticimportBaseModelfromhttp_noah.sync_clientimportSyncHTTPClientclassPet(BaseModel):name:strdefmain():withSyncHTTPClient("localhost",8080)asclient:pet:Pet=client.get("/pets/1",response_type=Pet)Let’s have a closer looks at what happened here:We provided onlyhostandportwithapi_basedefaulting to/api/v1so that
we don’t have to prepend it to every URL in our callWe ask http_noah to convert API response to an instance of the desired type (or raise
otherwise)We used a context manager to make sure everything will be cleaned up promptly.
In a more complex code, you may consider a kind of a life-cycle manager e.g. like in my demo
Hanuka project (source)Async example is pretty much the same:fromhttp_noah.async_clientimportAsyncHTTPClientasyncdefmain():asyncwithAsyncHTTPClient("localhost",8080)asclient:pet:Pet=awaitclient.get("/pets/1",response_type=Pet)Since the goal of this library is to provide similar interfaces for both sync and async
code I’ll focus onasyncexamples from now on and will be leaving notes if there are
differences that I worked hard to reduce to a very few.The client support the following methods that map the corresponding HTTP verbs:.get(...).post(...).put(...).delete(...)Sending your data back is easy as well - be it just a dict or Pydantic model.For Pydantic models you can just pass them to thebodyargument of e.g..post():asyncdefcreate_pet():asyncwithAsyncHTTPClient("localhost",8080)asclient:pet=Pet(name="Crispy")awaitclient.post("/pets",body=pet,response_type=Pet)If you just want to send data as JSON you need to outline that explicitly:fromhttp_noah.commonimportJSONDataasyncdefcreate_pet():asyncwithAsyncHTTPClient("localhost",8080)asclient:pet={"name":"Crispy"}awaitclient.post("/pets",body=JSONData(data=pet),response_type=Pet)This is necessary for http_noah to understand whether your intent is to send you data
as JSON or as Form which both can be Python dicts. See more on forms and file uploads
in the dedicated section below.Again, I prefer to model everything I send and receive with Pydantic models - it makes
life so much easier that you get addicted to it very fast.Nested ClientsNow when we understand the basic usage let’s see how can we build those beautiful nested
clients I promised you in the beginning.Let’s build a client for our hypothetical pet sanctuary API by starting with the root class:from__future__importannotationsfromhttp_noah.async_clientimportAsyncAPIClientBase,AsyncHTTPClientclassPetSanctuaryClient(AsyncAPIClientBase):@classmethoddefnew(cls,host:str,port:int,scheme:str="https")->PetSanctuaryClient:client=AsyncHTTPClient(host=host,port=port,scheme=scheme)returncls(client=client)A this point it’s just a boilerplate class that does nothing spectacular except having
a builder function. Note that I useAsyncAPIClientBaseand notAsyncHTTPClient.Now let’s implement Pets sub-API:from__future__importannotationsfromdataclassesimportdataclassfromhttp_noah.async_clientimportAsyncAPIClientBase,AsyncHTTPClient# Skipped model definitions here - as in the basic example@dataclassclassPetClient:client:AsyncHTTPClientclasspaths:prefix:str="/pets"list:str=prefixget:str=prefix+"/{id}"create:str=prefixasyncdeflist(self)->Pets:returnawaitself.client.get(self.paths.list,response_type=Pets)asyncdefget(self,id:int)->Pet:returnawaitself.client.get(self.paths.get.format(id=id),response_type=Pet)asyncdefcreate(self,pet:Pet)->Pet:returnawaitself.client.post(self.paths.create,body=Pet,response_type=Pet)@dataclassclassPetSanctuaryClient(AsyncAPIClientBase):pets:PetClient@classmethoddefnew(cls,host:str,port:int,scheme:str="https")->PetSanctuaryClient:client=AsyncHTTPClient(host=host,port=port,scheme=scheme)pet_client=PetClient(client)returncls(client=client,pets=pet_client)Now we are talking! Let’s enjoy it:psc=PetSanctuaryClient("localhost",8080,scheme="http")asyncwithpsc:pets=awaitpsc.pets.list()pet=awaitpsc.pets.get(1)Similarly we can implement other sub-API clients and nest them easily.Getting seriousResponse typeSpecifying response type ismandatoryunlessyou expect your request to respond with
HTTP 204 “No Content” which generally makes sense for DELETE operations.If response Content-Type heading is set toapplicaiton/jsonthen JSON data will be
decoded for you and can be further parsed usingPydanticmodel of your choice.Otherwise, you can request back eitherstrorbytesThis results in a limitation where with this library you can’t fetch JSON response back
as string. But since this is a high-level REST client I’ve yet bumped into this limitation
in practice.To sum it up, here are your options for theresponse_typeargument:byteswhen a request returns a binary data, e.g imagestrwhen a request returns text (technically speaking “when the content type is notapplication/json”)dict,list,int,bool,float,str(i.e. any of the JSON -> Python native types),
when your request returns JSON data and you don’t want it parsed further into Pydantic objects.Error handlingTrying to align between sync and async code I aliased common error base classes under
common namesConnectionError,HTTPError, andTimeoutErrorin bothhttp_noah.sync_clientandasync_client. This is where it stops though - behind the
name these are stillrequests/aiohttperror classes if you want to dig deeper.One useful thing that http_noah does for you is making sure to log HTTP body when the error occurs.
This is usually a small but vital piece of information to help you understand what’s going
on. Sadly enough, it requires quite a bit of tinkering to dig this info out.
Just one example is that calling aiohttp’s response objectraise_for_status()method
will actually return the underlying HTTP connection back to the pool depriving you of reading
the error body.Again, http_noah will log HTTP (error) body when it encounters HTTP errors.TimeoutsTimeouts can be configured by passing instance ofhttp_noah.common.Timeoutclass to
either.get(),put(), etc. methods or setting it per client instance throughClientOptions:fromhttp_noah.commonimportClientOptions,Timeoutfromhttp_noah.async_clientimportAsyncHTTPClientoptions=ClientOptions(Timeout(total=10)asyncwithAsyncHTTPClient(host="localhost",port=80,options=options)asclient:awaitclient.get(...)# Limited to 10 secondsawaitclient.post(...,timeout=Timeout(total=20))# per call overrideHowever, if you reflect on the nested client approach as was suggested earlier, you can quickly notice
that re-definingtimeoutargument in all your high-level methods is very onerous.
Fortunately, http_noah stands true to its name and provides an easy solution with
the help oftimeoutcontext manager that both sync and async client implements:Continuing ourPetSanctuaryClientexample:fromhttp_noah.commonimportTimeoutasyncwithPetSanctuaryClient("localhost",8080,scheme="http")aspsc:pets=awaitpsc.pets.list()withpsc.client.timeout(Timeout(total=1):pet=awaitpsc.pets.get(1)# Limited to 1 secondAs you can see, neitherPetClientnorPetSanctuaryClientdefined any timeout
logic yet we can perfectly apply timeouts.NoteOne difference between sync and async behaviour here is that in case of connection
timeout, aiohttp will raiseasync.TimeoutErrorwhere requests will raiserequests.exceptions.ConnectionErrorwhich is technically not a TimeoutError.Seetest_connect_timeouttests undertests/async_tests.pyandtests/sync_tests.pyfor details.FormsForms are not used much today. However, I still encounter them when I need to login
into API to get Bearer token.To use a form with http_noah simply fill it up as adict, as you would with
aiohttp / requests, and pass it throughbodyargument wrapped withFormData:fromtypingimportLiteralfrompydanticimportBaseModelfromhttp_noah.commonimportFormDataclassTokenResponse(BaseModel):access_token:strtoken_type:Literal["bearer"]asyncdefget_access_token():login_form=FormData(data={"grant_type":"password","username":"foo","password":"secret",})asyncwithAsyncHTTPClient("localhost",8080)asclient:tr=awaitclient.post("/access_token",body=login_form,response_type=TokenResponse)Fileshttp-noah provides simple means to upload a file as a multipart encoded form.
Best illustrated by example:frompathlibimportPathfromhttp_noah.commonimportUploadFileasyncwithAsyncHTTPClient("localhost",8080)asclient:awaitclient.post("/pets/1/photo",body=UploadFile(name="thumbnail",path=Path("myphoto.jpg"),)SSLSSL/TLS are supported as they are inrequestsandaiohttp. Sometimes however
it’s desirable to disable SSL validation, e.g. in your dev environment. This can be
done throughClientOptions:fromhttp_noah.commonimportClientOptionsfromhttp_noah.async_clientimportAsyncHTTPClientoptions=ClientOptions(ssl_verify_cert=False)asyncwithAsyncHTTPClient(host="localhost",port=80,options=options)asclient:...Authenticationhttp-noah support both Basic and Bearer token client authentication.
These can be set at any time on the existing client:asyncwithAsyncHTTPClient("localhost",8080)asclient:# Bearer tokenclient.set_auth_token("my-secret-token")# Or Basic Authclient.set_auth_basic("my-username","my-password")It’s a deliberate design decision to omit auth parameters from constructor
because in case of, e.g. bearer token, auth info may not be known in advance
because one may need to submit a login form first. Hence it’s required to be
able to set auth info at a later stage.DevelopmentTo develop http_noah you’ll need Python 3.8+, pipenv anddirenvinstalled.Then just runmake bootstrapafter cloning the repo, wait a while, and you are done - next time you enter into the
cloned directory the environment will be set for you.Code wise, you can’t really have the same code that does both sync and async. Not in a readable way at least.
Since readability counts and simplicity trumps complexity, I’d rather have two versions of a very simple code
that does each of sync and async instead of one callback-polluted/iterator-based/black-magic-imbued code-base.Care was takes to have a functional tests for each of the library features.Enjoy and see you at PRs!