Package: rest-collection-client
REST collectionlibrary client.This package provides bindings for http REST client creation.
This package has no complete solutions for http client, because it has much
application-specific things usually.HTTP client creation.Package provides 4 abstract classes for HTTP client:AbstractHttpClient- HTTP client skeleton.AbstractGetHttpClient- HTTP client for GET requests only skeleton.AbstractChunkedGetHttpClient- HTTP client for GET requests, which
request data with chunks (for smaller responses and network performance).AbstractAuthenticatedChunkedGetHttpClient- previos HTTP client, which
can send authorization requests before data requests.Example of http client## client.pyfrommathimportceilfromrest_collection_client.httpimport\AbstractAuthenticatedChunkedGetHttpClientdef_build_chunk_url(url,start,stop):"""Some code, that build url."""def_get_chunk_total(chunk):"""Some code, that get total value."""def_make_request(authentication_data):"""Some code, that make JSON-like data, that will be sended to server."""def_store_authorization_data(session,response):"""Some code, that store authorization data to session for next requests."""classRestCollectionClient(AbstractAuthenticatedChunkedGetHttpClient):def_compose_other_chunk_urls(self,url,chunk_size,first_chunk):total=_get_chunk_total(first_chunk)forindexinrange(1,ceil(total/chunk_size)):start=index*chunk_sizestop=(index+1)*chunk_sizeyield_build_chunk_url(url,start,stop)def_compose_first_chunk_url(self,url,chunk_size):return_build_chunk_url(url,0,chunk_size)asyncdef_request_authentication(self,authentication_data):url=authentication_data['url']request=_make_request(authentication_data)asyncwithself._session.post(url,json=request)asresp:ifresp.status!=200:returnFalse_store_authorization_data(self._session,resp)returnTrueUsage of this class:## fetch.pyfromasyncioimportget_event_loopfrom.clientimportRestCollectionClientchunk_size=1000authentication_data={'url':'http://example.com/login','login':'user','password':'secret_password'}data_url='http://example.com/data'asyncdefget():asyncwithRestCollectionClient()asclient:returnawaitclient.get(data_url,authentication_data,chunk_size=chunk_size)# we will request data here with chunks and authentication request before.deffetch_data():loop=get_event_loop()raw_response=loop.run_until_complete(get())loop.close()returnraw_responseJoin REST response in SQL-like manner.Data from REST collection API looks like this structure:{"a":[{"id":1,"name":"a"}],"b":[{"id":2,"a_id":1,"name":"b"}],"c":[{"id":2,"b_id":2,"name":"c"}]}They are normalized, like tables in SQL database. But, oftenly, we want to
build 2-dimensioned table from them.In SQL databases it is possible, when we use JOIN expression. So, we suggest
to use similar way, built on the top of famous library calledpandas.Declare join rulesFirst of all, we need to declare how our data relates internally. For this
purpose, this package provides some containers.## join.pyfromrest_collection_client.joinimportRestCollectionJoinColumns,\RestCollectionJoinRule,RestCollectionJoinSequenceRESPONSE_JOIN_SEQUENCE=RestCollectionJoinSequence(RestCollectionJoinRule(RestCollectionJoinColumns('a','id'),RestCollectionJoinColumns('b','a_id')),RestCollectionJoinRule(RestCollectionJoinColumns('b','id'),RestCollectionJoinColumns('c','b_id')),)# we will get 2d-table heredefmake_table(response):# response must be Mapping object with model names as keys and# `pandas.DataFrame` objects as values.returnRESPONSE_JOIN_SEQUENCE.join(response)In this example, we declare what columns of one model points to some columns
in another model for each bond. Our relations contains in each case only one
column pointing to one column, but this package supports multiple columns
declaration for each case as well.Prepare raw response data for joining.We know, that JSON accepts only simple types of data, like booleans, strings,
numbers. But we have to transfer more complex data sometimes, like
datetimes, timedeltas, decimals and others.So, we have to deserialize our raw response from server and convert them to
the map ofpandas.DataFrame. This packages provides some containers to
resolve this issue.## deserialize.pyfromrest_collection_client.responseimport\RestCollectionDeserializeMap,RestCollectionResponsedef_deserialize_a(a_item):passdef_deserialize_b(b_item):passdef_deserialize_c(c_item):passRESPONSE_DESERIALIZE_MAP=RestCollectionDeserializeMap(a=_deserialize_a,b=_deserialize_b,c=_deserialize_c)defdeserialize_response(raw_response):returnRestCollectionResponse.deserialize(raw_response,RESPONSE_DESERIALIZE_MAP)RestCollectionDeserializeMapcreates map of callables for each model, which
is required forRestCollectionResponse.deserializeclassmethod.RestCollectionResponseis aMappingobject, with model names as keys andpandas.DataFrameobjects as values (as join containers required).Rename response model columns.As we see, some models may have equal-named columns, like all models in our
example - we talk about "name" column. If we join our response in
2d-table, we will catch 3 columns, that will be called "name". It is very unclear, we can not predict what column belongs to some concrete model.Even worse case, if equal-named columns will appear in "join" column. We can
join out data in wrong way.That`s why,RestCollectionJoinSequencecontainer requires, that all column
names must be unique.By default,RestCollectionJoinColumnsconcatenates model name and column
name with dot, like this:a.name,b.a_id, etc... According to this, we
should rename columns in ourRestCollectionResponsebefore it will be
joined byRestCollectionJoinSequence.This package has default rename factory:## rename.pyfromrest_collection_client.responseimportRestCollectionRenameFactoryRESPONSE_RENAME_FACTORY=RestCollectionRenameFactory()defrename_response_values(response):returnresponse.rename(RESPONSE_RENAME_FACTORY)You can inheritRestCollectionRenameFactoryand implement your own factory,
but it must be related to rename manner ofRestCollectionJoinColumns.Both containers have keyword parameterdelimiter, to declare concatenate
delimiter.Full usage examplefrom.fetchimportfetch_datafrom.joinimportmake_tablefrom.deserializeimportdeserialize_responsefrom.renameimportrename_response_valuesdefmain():raw_response=fetch_data()response=deserialize_response(raw_response)response=rename_response_values(response)returnmake_table(response)After callingmainfunction, we will get joinedpandas.DataFramestructure with requested data.