Package: coveo-functools
coveo-functoolsIntrospection, finalizers, delegates, dispatchers, waiters...
These utilities aim at increasing productivity.annotationsIntrospect classes and callables at runtime.Can convert string annotations into their actual type reference.flexOverviewFlex works with annotations to adjust and convert input data to match your target structure.It was originally done as a mean to fitCamelCasepayloads from external APIs intosnake_caseclasses.Take for example this payload that we'd like to fit into a pep8 context:[{"Name":"John","SocialNumber":123},{"Name":"Jean","SocialNumber":123}]Explicit usage example:fromcoveo_functoolsimportflex@dataclassclassPerson:name:strsocial_number:Optional[int]=None# the deserializer is used directly to receive a list of Person instancesresponse=flex.deserialize(json.load(),hint=List[Person],errors='raise')Automatic usage example:fromcoveo_functools.fleximportflex@flex@dataclassclassSomeObject:""" I am decorated with @flex, so you can always give me some trouble. """name:strresponse=[SomeObject(**data)fordatainjson.load()]When remapping keys, Flex will ignore:CasingUnderscoresHyphensDotsSpacesFor instance, it will happily accept{"__NaM e._": "John"}as valid input for thePersonclass.It can also create instances of custom classes:@dataclassclassAddress:street:str@dataclassclassPerson:name:straddress:List[Address]social_number:Optional[int]=NoneYou could then feed it a payload like{"name": "Lucy", address: [{"street": ...}, {...}]}.
Flex will create an instance ofPerson, that has a list of 2 Address instances.Note: The basic typesstr, bool, int, float, dict, list, Noneare ignored(no conversion occurs).
This is becausejson.load()already returns these values in the proper type. This may change in the future.Supported objects and annotationsFlex can be used with:Classes and dataclassesAbstract classes(new in 2.0.9)(requires adapter or serialization; explained below)Enums(new in 2.0.6)(Enum data-type support added in 2.0.26)Literals(new in 2.0.21)FunctionsMethodsUnion[str, bool, int, float, list, dict, None](or any combination of these basicjson-compatible types)These typing constructs, whereTis your custom class:List[T]Dict[str, T](in 2.0.23 and below)Dict[K, V](new in 2.0.24; dict keys are no longer limited to strings)Union[T, List[T]](for APIs that may return a thing-or-list-of-things)Optional[T]InitVar[T](new in 2.0.25)LimitationsVariable positional args (such asdef fn(*args): ...) are left untouched.Basic json-compatible types will be left untouched. This is determined by the annotation, not the actual value.IfNoneis given as a value to deserialize into anything,Noneis given back. Absolutely no validation occurs in this case.An Abstract class requires additional metadata or a subclass adapter.No support for additionaltypingandcollectionsobjects other than the ones mentioned in this documentation.You can onlyUnionbasic json-compatible types, orList[T], T.All referenced types must be importable from the module it is defined in. This means that you cannot use inline and dynamic classes.These are subject to change.Features and FAQsSubclass Adaptersnew in 2.0.9You can direct and override how to instantiate a payload by registering a callback adapter.The adapter is aCallable[[Any], TypeHint]that you provide.
It will be called with the payload value asAny, so you can inspect the content.
It must return aTypeHintthat tells flex which class to use.new in 2.0.10: TypeHint may also be a callable (previously, it had to be a class).With subclass adapters, you can selectively decide the implementation class based on the payload to deserialize.
While this is necessary when annotating structures with Abstract classes,
it can be used for any other class as well.For this to work, you must register the annotated class with a callback:fromcoveo_functools.flex.subclass_adapterimportregister_subclass_adapterclassAbstract:@abstractmethoddefapi(self)->None:...classThisImplementation(Abstract):defapi(self)->None:...classOtherImplementation(Abstract):defapi(self)->None:...defadapter(payload:Any)->Type:assertisinstance(payload,dict)# actual type depends on payloadreturnThisImplementationif'this'inpayloadelseOtherImplementationregister_subclass_adapter(Abstract,adapter)Thanks to the adapter, this is now possible:assertisinstance(deserialize({'this':{}},hint=Abstract),ThisImplementation)assertisinstance(deserialize({},hint=Abstract),OtherImplementation)@dataclassclassPayload:owner:Abstractinstance=deserialize({"owner":{"this":{}}},hint=Payload)assertisinstance(instance.owner,ThisImplementation)The intended use of subclass adapters is to:Support Abstract classes as annotationsBeing able to specify a delegate for a specific kind of payloadTo enhance/clean a payload before it is usedAny other use will generally:Mess up your type annotation game because types are altered dynamically at runtime.Make your code more obscure and more likely to investigate the dark arts.Break your IDE's autocompletion features.Linters which rely on static analysis will not be as powerful as they could be.That being said, it's a powerful and potentially game-breaking feature
that can be used to bend the framework if you accept bearing the consequences:There are no validations (to allow duck typing and stuff)This means you don'thaveto return an actual subclass; just something that can handle that payloadYou can register a callback forAny(or anything else really)You're not limited to return custom classes: you can return things likeDict[str, int]orList[Implementation]and the flex machinery will handle it just as if it was statically annotated that way.The payloadvaluereceived by the adapter is not a copy, modifications will be honored.Factory AdaptersSome types don't play well with keyword arguments. For instance, using thedatetimeclass is much more convenient using strings, isoformat() and fromisoformat() than
having to parse it into the year/month/etc component.To serialize such types, you can use a factory adapter, which is expected to return the instance instead of the type.fromdataclassesimportdataclassfromdatetimeimportdatetimefromcoveo_functoolsimportflexfromcoveo_functools.flex.factory_adapterimportregister_factory_adapter# Implementation of the deserialization:def_datetime_factory(value:str)->datetime:returndatetime.fromisoformat(value)register_factory_adapter(datetime,_datetime_factory)# Demonstration:@dataclassclassWithDateTime:timestamp:datetimetimestamp=datetime.utcnow()instance=flex.deserialize({"timestamp":timestamp.isoformat()},hint=WithDateTime,errors='raise')assertinstance.timestamp==timestampassertisinstance(instance.timestamp,datetime)About Abstract classesThere are two ways to deal with abstract classes:If you control the serialization aspect, flex can inspect a custom instance and generate metadata information that can be used during the deserialization process.This is the way.If you don't control the serialization (e.g.: it's a json payload from an api), you can attach callbacks to inspect the payload and return the non-abstract class to use.Abstract using SerializationMetadatanew in 2.0.10Note: This is a dumbed down / magic version of the functionality offered bypickleandyaml,
with the difference that the metadata and the payload are kept separate and readable.TheSerializationMetadataclass inspects an instance and stores the type of the objects within (not their value!).
Think of it as a header that must accompany your serialized data, so you can rebuild it later using the same subclasses.This allows you use abstract classes in annotations, but deserialize into concrete ones:fromcoveo_functools.flex.deserializerimportdeserializefromcoveo_functools.flex.serializerimportSerializationMetadataclassAbstract:@abstractmethoddefapi(self)->None:...@dataclassclassConcrete(Abstract):defapi(self)->None:...@dataclassclassParent:nested:Optional[Abstract]=Nonemeta=SerializationMetadata.from_instance(Parent(Concrete()))parent=deserialize({"nested":{}},hint=meta,errors='raise')assertisinstance(parent.nested,Concrete)Serialization metadata ties to a payload/instance and none other.
It keeps the type information of your instance, and thus may be different on a different instance.
For each payload you want to store, you must generate a newSerializationMetadatainstance.Example use cases:Serialize objects into a cloud-based queueStore objects into a no-SQL databaseLimitation: The concrete implementation must be importable using the normal python mechanisms.
A class returned from a function is not importable, and will not deserialize correctly.Abstract using Subclass adaptersnew in 2.0.9The other, more involved way to use Abstract classes as annotations is to register subclass adapters.See theSubclass Adapterssection for more info.About EnumsEnums will resolve by value or name, in this order:By exact value (str/int/etc)By exact name (str)By flexed value (str)By flexed name (str)flex.deserializeThis is where the magic happens, and is the recommended usage whenever it meets your use case.TL;DR: Given thatpayloadis a dict,flex.deserialize(payload, hint=Job)will convertpayloadinto an instance ofJob.Here's an example puzzle! An uncanny API returns a messy "transaction" JSON:{"Sold_To":{"Name":"Jon"},"Items":[{"sku":123,"price":19.99},{"sku":234,"price":13.99},{"sku":0,"price":0.50,"NOTE":"Forgot the reusable bag at home!!"}],"Rebates":{"airmiles":{"Flat":10.0},"coupon":[{"Flat":0.79},{"Flat":1.50}],"senior":{"Percentage":2.5}},"Id":"GgfhAs89876yh.z"}Wouldn't it be convenient if you could create simple classes/dataclasses around them without any boilerplate?You can solve it with flex. In one line, too!Start by designing a hierarchy of classes with annotations that closely follow the API reference.Remember, casing and underscore are ignored in flex, so you could use pep8 if you want:# models.pyfromdataclassesimportdataclassfromtypingimportList,Dict,Union,OptionalclassSkuItem:def__init__(self,sku:int,price:float)->None:self.sku=skuself.price=price@dataclassclassRebate:percentage:Optional[float]=Noneflat:Optional[float]=None@dataclassclassCustomer:name:str@dataclassclassTransaction:sold_to:Customeritems:List[SkuItem]rebates:Dict[str,Union[Rebate,List[Rebate]]]Did you notice any flex-related boilerplate in the snippet above? No? Good! :)Here's how you can use the flex deserializer to bend the furious API response into your perfect python classes:payload={"Sold_To":{"Name":"Jon"},"Items":[{"sku":123,"price":19.99},{"sku":234,"price":13.99},{"sku":0,"price":0.50,"NOTE":"Forgot the reusable bag at home!!"}],"Rebates":{"airmiles":{"Flat":10.0},"coupon":[{"Flat":0.79},{"Flat":1.50}],"senior":{"Percentage":2.5}},"Id":"GgfhAs89876yh.z"}transaction=flex.deserialize(payload,hint=Transaction,errors='raise')all_transactions=flex.deserialize([payload,payload],hint=List[Transaction],errors='raise')Interesting details:Well, the casing worked! :shrug:IdandNOTEwere dropped because they were excluded from theTransactionmodel. Time saver; some APIs returntonsof data.The rebates actually kept the keys, and createdRebateinstances as the values.The value type of therebatesdict is either a singleRebateinstance or a list of them. See the "thing or list of things" section below for considerations.In theall_transactionsvariable,List[Annotation]was used directly as the hint. Nifty!@flexandflex(obj)There is a decorator version ofdeserialize.from coveo_functools.flex import flexIt returns a function, method or class wrapped inflex.deserializemagic.
When called, the wrapper will automatically adjust the call arguments to match the wrapped object, call the wrapped object with them, and return the response.flexcan be used:as a decorator over classes, methods and functionsinline to call a function or to create flexible factoriesWhen used inline, you can adjust a payload for any callable:fromsome_3rd_partyimportcalculate_priceprice=flex(calculate_price)(**payload)You can also generate "flexible" factories, for instance to be used as a delegate:fromsome_3rd_partyimportThisClassfactory:Callable[...,T]=flex(ThisClass)instance1=factory(**payload1)instance2=factory(**payload2)When used as a decorator, all invocations are automatically handled for all callers:@flexdefcalculate_price(sold_to:Customer,items:Union[SkuItems,List[SkuItems]])->float:...# breaks static analysis; wrong argument shown for demonstration purposesprice=calculate_price(SoldTo=dict(Name="Marie"),items={"sku":123,"price":19.99})You could adjust theTransactionfrom earlier class like this:@flex@dataclassclassTransaction:sold_to:Customeritems:List[SkuItem]rebates:Dict[str,Union[Rebate,List[Rebate]]]So that you can drop the explicit calls toflex.deserializeand use them directly:one_transaction=Transaction(**payload)list_transactions=[Transaction(**t)fortin[payload,payload]]flexordeserialize?Favorflex.deserializeover the decorator pattern:This will make the usages explicit rather than implicit.The additional wrappers created by the decorator may affect performance in the presence of huge structures.You canflex.deserialize([], hint=List[T])and get a list, but you cannotflex(List[T])directly (both methods demonstrated below)Generally, it leads to a better design because you end up wiring theflex.deserializecall next to thejson.load()call in a generic manner, and that's 100% of theflexcode you'll ever need:classApiWrapper:defget_transaction(self,id:int)->Transaction:returnself._do_request("GET",f"api/transactions/{id}",hint=Transaction)defget_all_transactions(self)->List[Transaction]:returnself._do_request("GET","api/transactions",hint=List[Transaction])def_do_request(self,method:str,url:str,hint:T)->T:response=self._session.request(method=method,url=url)returnflex.deserialize(response.json,hint=hint)Because explicit is better than implicit, the above design is generally easier to understand than the one below, whereTransactionis assumed to be decorated with@flex:classApiWrapper:defget_transaction(self,id:int)->Transaction:returnTransaction(**self._do_request("GET",f"api/transactions/{id}"))defget_all_transactions(self)->List[Transaction]:return[Transaction(**data)fordatainself._do_request("GET","api/transactions")]def_do_request(self,method:str,url:str)->Any:response=self._session.request(method=method,url=url)returnresponse.jsonConsideration for mypyThere is one annotation case worth mentioning.
Consider this code:classInner:...@flexdeffn(inner:Inner)->...:..._=fn(**{'inner':{...}})In this case, mypy will infer that you're doing**Dict[str, Dict]and complain that Dict is not compatible with Inner.To solve this without an ignore statement,
explicitly annotate/cast your payloads with Any:payload:Dict[str,Any]={"inner":{}}_=fn(**payload)unflexUnflex is one of the utilities used byflex.deserializer.It is responsible for adjusting the keyword arguments of a dictionary, so that they match the argument names of a target function.It does not perform any conversion; all it does is matching keys.
Extra keys are dropped by default:fromcoveo_functools.fleximportunflexdeffn(arg1:str,arg2:str)->None:...assertunflex(fn,{"ARG1":...,"ArG_2":...,"extra":...})=={"arg1":...,"arg2":...}Note: To target classes, you need tounflex(cls.__init__, ...)@flexcaseflexcaseis the decorator version ofunflex:fromcoveo_functools.fleximportflexcase@flexcasedeffn(arg1:str,arg2:str)->str:returnf"{arg1}{arg2}"assertfn(ARG1="hello",_arg2="world",extra=...)=="hello world"dispatchAn enhanced version offunctools.singledispatch:Adds support forType[]annotations (singledispatch only works on instances)You are no longer limited to the first argument of the methodYou can target an argument by its name too, regardless of its positionfinalizerA classic and simple try/finally context manager that launches a delegate once a block of code has completed.A common trick is to "cook" the finalizer arguments through a mutable type such as a list or dict:fromtypingimportListfromcoveo_functools.finalizerimportfinalizerdefclean_up(container_names:List[str])->None:for_incontainer_names:...deftest_spawning_containers()->None:containers:List[str]=[]withfinalizer(clean_up,containers):containers.append('some-container-1')containers.append('some-container-2')containers.append('some-container-3')wait.until()Waits for a condition to happen. Can be configured with exceptions to ignore.fromcoveo_functoolsimportwaitimportrequestsdef_ready()->bool:returnrequests.get('/ping').status_code==200wait.until(_ready,timeout_s=30,retry_ms=100,handle_exceptions=ConnectionError,failure_message="The service failed to respond in time.")wait.BackoffA customizable class to assist in the creation of backoff retry strategies.Customizable growth factorJitterBackoff progress % (want to fire some preliminary alarms at 50% backoff maybe?)Supports infinite backoffCan be configured to raise after too many attemptsCan be configured to raise after a set amount of timee.g.: Worker loop failure management by catching RetriesExhaustedfromcoveo_functools.waitimportBackoffbackoff=Backoff()whilemy_loop:try:do_stuff()exceptExceptionasexception:try:quit_flag.wait(next(backoff))exceptbackoff.RetriesExhausted:raiseexceptione.g.: Worker loop failure management without the nested try/catch:fromcoveo_functools.waitimportBackoffbackoff=Backoff()whilemy_loop:try:do_stuff()exceptExceptionasexception:wait_time=next(backoff,None)ifwait_timeisNone:raiseexceptionquit_flag.wait(wait_time)e.g.: You can generate the wait times without creating a Backoff instance, too:importtimefromcoveo_functools.waitimportBackoffwait_times=list(Backoff.generate_backoff_stages(first_wait,growth,max_backoff))forsleep_timeinwait_times:try:do_stuff()breakexcept:time.sleep(sleep_time)else:raiseImSickOfTrying()