Package: pypwext
PyPwExtThis is a extension, decorators and other utilities that complements thewAWS Lambda Powertoolslibrary. It is centered around the following fourpillarsLogginglogger=PyPwExtLogger()@logger.method(classification=InfoClassification.PII)defsubscribe(customer:Customer)->Subscription:...Error handlingerrors=PyPwExtErrorHandler()@errors.collectdefdo_operation():raisePyPwExtHTTPError('Something went wrong',details={'foo':'bar'})HTTP and Lambda Communicationhttp=PyPwExtHTTPSession()# Defaults with "sane" retries, exp back-off etc.@http.method(method='POST',url='https://{STAGE}.execute-api.{AWS_REGION}.amazonaws.com/{api_version}/my-api'params={'email':'{email}'},body='customer')defsend_offer(api_version:str,email:str,customer:Customer,response_code:HTTPStatus=HTTPStatus.OK,response_body:str='')->str:...µService Supportservice=PyPwExtService()@service.response@event_parser(model=Order)defhandler(event:Order,context:LambdaContext):ifnotevent.pypwext_id:raiseStdPyPwExtError(code=HTTPStatus.BAD_REQUEST,message="Missing pypwext_id",)returnPyPwExtResponse(status_code=HTTPStatus.OK,updated=[itemforiteminevent.items],operation="create-order",)OverviewThis module contains the core functionality of the PyPwExt µService applications.:bulb: Each piece of functionality is implemented as an individual function and isnotdependant on each other. Hence, it is up to the implementor tochoosewhich ones to use.It consists ofstructuralandsemanticelements to align theµServiceswhen logging,
error, input and output handling. It doesnotmandate any specific model when it comes to
the content of the payloads sent back and forth.The onlyreservedreturn body element iserror. But that is if you usePyPwExtResponseandPyPwExtService.responsedecorator. You may even omit that if you set the@response(just_status_code=True), and that is thedefaultbehavior 😊For example:service=PyPwExtService()logger=PyPwExtLogger(service='my-service')@logger.method(classification=InfoClassification.PII)@service.responsedefmy_service(request):raisePyPwExtHTTPError(message='bad input',details={Operation:'my-operation''org_no':'7112234455'})By default, this would return the followingHTTP status code: 400Body: NoneIf you instead set@service.response(just_status_code=False)it would return{"error":{"message":"bad input","details":{"operation":"my-operation","org_no":"7112234455"}}}A extended exampleThe example below uses most of the features of thePyPwExtmain types. Even though each feature supports quite a few configuration options to fit the exact purpose, it comes with many"sane"defaults. Hence, many variations can be avoided. But it is there when it is needed.logger=PyPwExtLogger(default_logger=True,service='my-service')errors=PyPwExtErrorHandler()service=PyPwExtService()http=PyPwExtHTTPSession()# Defaults with "sane" retries, exp back-off etc.classCustomer(BaseMode,extra=Extra.allow)email:strage:Optional[int]=NoneclassCustomers(BaseModel,extra=Extra.allow):Customers:List[Customer]=Field(default_factory=list)@errors.collect@logger.method(log_exception=False)@http.method(method='POST',url='https://{STAGE}.execute-api.{AWS_REGION}.amazonaws.com/{api_version}/my-api'params={'email':'{email}'},body='customer')defsend_offer(api_version:str,email:str,customer:Customer,response_code:HTTPStatus=HTTPStatus.OK,response_body:str='')->str:# send offer to customer -> fail -> raiseifresponse_code==HTTPStatus.OK:raisePyPwExtErrorWithReturn(code=HTTPStatus.BAD_REQUEST,message=f"Failed to send offer to customer:{email}",details={Operation:'send_offer','customer':customer})returnemail@logger.method(type=LogEntryType.AUDIT,operation='send_offer')@errors.collect(root=True)@service.response@event_parser(model=CustomerData)defhandler(event:CustomerData,ctx:LambdaContext)->any:returnPyPwExtResponse(status_code=HTTPStatus.OK,updated=list(filter(partial(is_not,None),[send_offer('1.0.0',c.email,c)forcinevent])),operation="create-offer")Since,@service.responseby default returns the "largest" status code (otherwisecode_from_error=False). The above could, for example, produce the following response payload:{"updated":["nisse@manpower.com"],"operation":"create-offer","error":[{"code":400,"msg":"Failed to find record for customer: mario.toffia@pypwext.se","classification":"NA","details":{"operation":"send_offer","customer":{"email":"mario.toffia@pypwext.se","org_no":"1234567890"}}},{"code":400,"msg":"Failed to find record for customer: ivar@ikea.se","classification":"NA","details":{"operation":"send_offer","customer":{"email":"ivar@ikea.se","org_no":"0987654321"}}}]}Since@logger.methodis applied, it logs it usingDEBUGverbosity at the entry, andINFOlog the exit by default. However, it won't log the exception in thesend_offerfunction since it would fill the log with cluttered information.The@http.methodautomatically detectsAWS APIendpoints and usesSigV4by default. However, it is completely configurable in thePyPwExtHTTPSessionconstructor.Since no argument given in@service.response, thePyPwExtResponseobject is converted to an API Gateway response that supports eitherRESTorHTTPv2version of the API Gateway. Thus theJSONpayload is thebodypart of the response, and thestatusCodeis set to 402 (NOT FOUND) since that was the last collected error. Of course, you may override this behaviour if you want.ThePyPwExtalso comes with an out of the boxPyPwExtJSONEncodercapable of handling many object types and
is extensible. For example, it adheres tobasemodule protocols such asSupportsToJsonandSupportsToCuratedDict(that alsopydantic.BaseModelalso exposes).A note on the ordering of the decoratorsHow decorators wrap the functions is basicPythonknowledge. However, it is sometimes essential which order the function is decorated.fromaws_lambda_powertools.utilities.data_classesimport(event_source,APIGatewayProxyEventV2)@event_source(data_class=APIGatewayProxyEventV2)@logger.method@errors.collect(root=True)@service.response(just_status_code=False)defhandler(event:APIGatewayProxyEventV2,,context:LambdaContext):...First, install arootcollector so the@service.responsemay pick those errors and merge those with the response.
Therefore, the@errors.collectmust be before (thus installs the collector for the@service.responseto use).The intention of@logger.methodis to log theAPIGatewayProxyEventV2in addition to theResponseobject. It also logs
any raisedPyPwExtErrorobject.Then, for the@event_sourceto get its data, parse it to anAPIGatewayProxyEventV2object.:bulb:Hence, the decorator"execution"order is top-down and returns bottom-up. You decide what you want to happen!Sample Typed µServiceclassPyPwExtModel(BaseModel):# NOTE: You may use the @dataclass decorator instead of the BaseModel classpypwext_id:str"""Unique trace id, to be passed in all systems and logged to make trails"""classOrderItem(BaseModel):id:intquantity:intdescription:strclassOrder(PyPwExtModel):id:intdescription:stritems:List[OrderItem]optional_field:Optional[str]logger=PyPwExtLogger()metrics=Metrics()service=PyPwExtService()@service.response@logger.method@metrics.log_metrics@event_parser(model=Order)defhandler(event:Order,context:LambdaContext):ifnotevent.pypwext_id:raiseStdPyPwExtError(code=HTTPStatus.BAD_REQUEST,message="Missing pypwext_id",)returnPyPwExtResponse(status_code=HTTPStatus.OK,updated=[itemforiteminevent.items],operation="create-order",)The sample renders an API Gatewaybodyresponse of:{"operation":"create-order","updated":[{"id":1015938732,"quantity":1,"description":"item xpto"}]}It logs the objects in the payload and the response, and if any error occurs, if exception-log it automatically. It also records metrics formy-servicein the namespacemy-namespaceas an example of setting the environment variablesPOWERTOOLS_SERVICE_NAMEandPOWERTOOLS_METRICS_NAMESPACEtomy-serviceandmy-namespacerespectively.Sample Calling other HTTP endpointsTheHTTPmodule returns a pre-configured HTTP session that defaults. For example, it can configure the number of retries and the timeout and which methods and response codes should yield a retry.When retrying, it uses an exponential back-off and handles temporal outages.withPyPwExtHTTPSession(logger=logger)ashttp:response=http.get('https://api.openaq.org/v1/cities',params={'country':'SE'})Below, do reconfigure the timeout and logs on request and response. It also reconfigures the number of retries to 3 and a higher back-off factor (wait longer time).logger=PyPwExtLogger()withPyPwExtHTTPSession(PyPwExtHTTPAdapter(timeout=10,logger=logger),PyPwExtRetry(total=3,backoff_factor=2),)ashttp:response=http.get("https://en.wikipedia.org/w/api.php")If anAPI Gatewaycall is wanted. It is expected to be on the following form:https://{api-gateway-id}.execute-api.{region}.amazonaws.com/..... TheHTTPmodule automatically configures aBotoAWSRequestsAuthand set it to therequest.authparameter using theAWS_REGIONenvironment variable. Thus, this makes the request match aSigV4request and gets authenticated at theAPI Gatewayusing theAWS_ACCESS_KEY_IDandAWS_SECRET_ACCESS_KEY.withPyPwExtHTTPSession(region='eu-north-1')ashttp:http.get('https://abc123.execute-api.eu-north-1.amazonaws.com/dev/cities',params={'country':'SE'})The above example overrides theAWS_REGIONenvironment toeu-north-1.NOTE: Since thePyPwExtHTTPSessionis a standard python libraryHTTPSession, it pools connections and so on and thus should be
cached to avoid the overhead of creating a new session for each request.ThePyPwExtHTTPSessionextension handles synchronous and asynchronous lambda calls with configurable retries and "sensible" defaults. It is possible to use them from code or decorated on the function.The sample below invokes the lambda synchronously and passes custom parameters in theLambdaContextand nothing as the body. It has a default of 10 times before giving up.withPyPwExtHTTPSession()ashttp:result=http.func(url='mario-unit-test-function',params={'country':'SE'})HTTP Decorator SamplesIt is possible to decorate a function with theHTTPdecorator to make theHTTPcall and process the response in method or make the decorator return the response. The latter automatically raises aPyPwExtHTTPErrorif theHTTPstatus code is 299 or greater.The simplest one is to make the request declarative with no processing in function (i.e. not declare any of theresponse,response_bodyorresponse_codeparameters in the function prototype).http=PyPwExtHTTPSession()@http.method(url='https://{STAGE}.api.openaq.org/v1/cities',params={'country':'{country}'})defcities(country:str)->requests.Response:passtryresponse=cities(country='SE')# This will call the site and return the responseprint(response.text)exceptPyPwExtHTTPErrorase:print(e.response_body)It is possible to process the response in the function and manually handle errors. Supply with any of the "response" parameters. Make sure the declare the "response" parameters with defaults (e.g.response = None).http=PyPwExtHTTPSession()@http.method(url='https://api.openaq.org/v1/cities',params={'country':'{country}'})defcities(country:str,response_body:str='',response_code:HTTPStatus=HTTPStatus.NOT_FOUND)->str:ifresponse_code==http.HTTPStatus.OK:returnf'{country}has{response_body["count"]}cities'else:raisePyPwExtHTTPError(code=response_code,message=response_body)The URL is formatted with parameters supplied to the function whenUPPERCASEfinds those as environment variables. It then makes the
call and passes the response to the function. The function may then process the response and return a string.Since thePyPwExtHTTPSessionis by default resolving AWS API gateway calls, it uses theBotoAWSRequestsAuthto set the credentials by creating a SigV4 request (it is possible to turn off this behaviour).http=PyPwExtHTTPSession()@http.method(method='POST',url='https://{gw_id}.execute-api.{AWS_REGION}.amazonaws.com/dev/cities',body='city')defget_cities(gw_id:str,city:str,response:Optional[requests.Response]=None)->Dict[str,Any]:""" Gets the cities from a specific country.Args:gw_id:  The API Gateway IDcity:   The body containing the country to fetch.for example: `{"country": "SE"}`Returns:The response body."""return{'country':country,'result':json.loads(response.text)}value=get_cities('abc123','SE','the body')HTTP Decorator Lambda SamplesIt is possible to invoke the lambda function synchronously (FUNC) and asynchronously (EVENT). Given the following lambda functiondeflambda_handler(event,context):ifevent.get('country'):country=event['country']else:country=context.client_context.custom['country']# contrives to show that you can use custom data in the contextresult=requests.get('https://api.openaq.org/v1/cities',params={'country':country})ifresult.status_code!=200:raiseValueError(f'Failed to get cities result:{result.status_code}')returnjson.loads(result.text)Invoking the lambda usingFUNC(synchronous) is as simple as:@http.method(method='FUNC',url='arn:aws:lambda:eu-west-1:010711114025:function:mario-unit-test-function',params={'country':'{country}'})defget_cities(country:str,response:LambdaResponse=None)->str:ifresponse.StatusCode==HTTPStatus.OK.value:returnresponse.payload_as_text()else:raisePyPwExtHTTPError(code=response.StatusCode,message=f'Failed to get cities from{country}',details={'error':response.payload_as_text()})value=get_cities(country='SE'):bulb:NOTE:The lambda function is invoked synchronously. Hence, theresponseis aLambdaResponseobject instead of therequests.Responseobject. Therefore, thee returned payload from the lambda.If you'd like to invoke the lambda asynchronously, you can use theEVENTmethod. The following example is for a lambda function that is invoked asynchronously.@http.method(method='EVENT',url='{STAGE}-function:mario-unit-test-function',body='body')defget_cities(body:Dict[str,Any],response:LambdaResponse=None)->str:ifresponse.StatusCode==HTTPStatus.ACCEPTED.value:returnjson.dumps(response.ResponseMetadata)else:raisePyPwExtHTTPError(code=response.StatusCode,message=f'Failed to get cities from{body}',details={'error':response.FunctionError})value=get_cities({'country':'SE'})Since it is asynchronous, no customLambdaContextcan be used. Note that the successful invocation is indicated by theStatusCodebeing202(Accepted). Since the lambda cannot return anything, theLambdaResponse.Payloaddo not contain anything.It is still possible to use lambda'sresponse_bodyandresponse_codeparameters. Theresponse_bodyforEVENTtype will bejson.dumps(response.ResponseMetadata). When creating aPyPwExtHTTPSessionit is possible to override the PyPwExt defaults for the boto-coreConfigobject. The default is:Config(region_name=self.region,# region is either manual set or AWS_REGION env varconnect_timeout=60,read_timeout=60,retries={'total_max_attempts':10,'max_attempts':10,'mode':'adaptive'})If you supply aConfigobject, it is merged over the defaults.Sample logging outputThe logger, independent if it is decorated or used directly, forces few new fieldsleveltimestampmessageserviceclassificationtypeoperationAdditionally, if you enable log lambda context, it adds a set of function_* fields. If x-ray is enabled, it logs that as well. The below entry also adds acorrelation_idof the REST API Gateway request-id.@logger.inject_lambda_context(correlation_id_path=correlation_paths.API_GATEWAY_REST)defhandler(event,context):returnapp.resolve(event,context)Sample log entry:{"level":"ERROR","location":"my_func:555","message":{"msg":"Exception in my_func","args":{"person_ssn":"19120102-01921"}},"timestamp":"2021-11-24 13:38:50,673+0000","service":"my service","classification":"PII","type":"AUDIT","cold_start":true,"function_name":"test_func","function_memory_size":"1024","function_arn":"arn:aws:lambda:eu-west-1:010711114025:function:test_func","function_request_id":"a89efe17-7a9e-49f8-9b4c-f00abd7b2ac8","correlation_id":"1db848fb-bf8b-4b5d-b2ad-61e2175181b4","operation":"get-user-info","exception":"...exception...","exception_name":"StdPyPwExtError","xray_trace_id":"1-619e4067-62471735a4cf522db5720c7e"}PhilosophyThePyPwExtis a µService framework designed to beopt-ininstead ofopt-out. It means
that you may always choose to use the functionality instead ofREQUIREDto use it. So, for example, if you
have decorated ahandlerwith@pypwext_response, you always have the option to bypass it, e.g. returning
your own customJSONresponse.logger=PyPwExtLogger(service='my-service')service=PyPwExtService()@logger.method@service.responsedefhandler(customers:List[str]):returnjson.dumps({"statusCode":200,"body":{"updated":list(filter(partial(is_not,None),[send_offer(c)forcincustomers])),Operation:"create-offer"}})The above still logs if any error occurs and returns a standardized response upon exception but completely bypassing
response handling. Instead, it returns theJSONresponse as-is.You may use small parts or the whole package, and each decorator can be used separately. It is also possible to access parts
from code such as the@errors.collectdecorator do expose currently collected errors through.get_current_collector()where actions may be taken due to collected errors down the chain. Or you may choose to clear errors and let the decorator
process as those have never been collected.The important part ofPyPwExtis that the µServices are semantically and structurally the same. Therefore some
base types and reservedkeywordsdo exist. For example, when a particular operation is commenced, theOperationkey facades the value. For example:logger=PyPwExtLogger(service='my-service')@logger.method(operation='create-offer')defcreate_offer(...)->...:...The above logs the entry withDEBUGand exit withINFOlevel automatically with the{"operation": "create-offer"}.
It also uses standardized keywords such asReturnandArgumentsto log the entry arguments and return values.Using standardized keywords makes it easier to understand and search in the logs.Of course, the above could be done in code, but not recommended.logger.info({Operation:"pay",Classification:InfoClassification.CORPORATE_SENSITIVE_INFO,LogType:LogEntryType.AUDIT,Arguments:{"amount":amount,"credit_card_id":credit_card_id}})The above sample uses constants defined in thebasemodule. Since thePyPwExtLoggeris based on the Powertools logger, it is quite possible to, e.g. include@logger.inject_lambda_context(log_event=True)to log the Lambda event including lambda context variables automatically.DependenciesPyPwExt is heavily dependant on theAWS Lambda Powertoolswhere thePyPwExtLoggerderives from the PowertoolsLoggerandPyPwExtReturnunderstand API ProxyResponsenatively.An optional dependency that plugs right in isPydantic. It is a model and validation framework that can validate the input and output of the microservice. In addition, theAWS Lambda Powertoolsrelies onPydanticwhen using theTyping Module.Setup development environmentCreate a virtual python environmentpython -m venv .venv.Activate it by. .venv/bin/activate.Install the dependencies bymake dependencies.Now the environment is ready to use.