Package: asgarde
AsgardeThis module allows simplifying error handling with Apache Beam Python.Versions compatibility between Beam and AsgardeAsgardeBeam0.16.0>= 2.37.0Installation of projectThe project is hosted on PyPi repository.You can install it with all the build tools compatibles with PyPi and pip.PyPiExample with pip command line from bashpipinstallasgarde==0.16.0Example with requirements.txtrequirements.txt fileasgarde==0.16.0pipinstall-rrequirements.txtExample with PipenvPipFile[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
asgarde = "==0.16.0"

[requires]
python_version = "3.8"pipenvshell

pipenvinstallpipenv shell creates a virtual envpipenv installinstalls all the packages specified in the PipfileA PipFile.lock is generated with a hash on installed packageshttps://pipenv.pypa.io/en/latest/Example of native error handling with BeamThe following example shows error handling in each step with usual Beam code.@dataclassclassTeamInfo:name:strcountry:strcity:str@dataclassclassFailure:pipeline_step:strinput_element:strexception:Exceptionteam_names=['PSG','OL','Real','ManU']team_countries={'PSG':'France','OL':'France','Real':'Spain','ManU':'England'}team_cities={'PSG':'Paris','OL':'France','Real':'Madrid','ManU':'Manchester'}classMapToTeamWithCountry(DoFn):defprocess(self,element,*args,**kwargs):try:team_name:str=elementyieldTeamInfo(name=team_name,country=team_countries[team_name],city='')exceptExceptionaserr:failure=Failure(pipeline_step="Map 1",input_element=element,exception=err)yieldpvalue.TaggedOutput(FAILURES,failure)classMapToTeamWithCity(DoFn):defprocess(self,element,*args,**kwargs):try:team_info:TeamInfo=elementcity:str=team_cities[team_info.name]yieldTeamInfo(name=team_info.name,country=team_info.country,city=city)exceptExceptionaserr:failure=Failure(pipeline_step="Map 2",input_element=element,exception=err)yieldpvalue.TaggedOutput(FAILURES,failure)classFilterFranceTeams(DoFn):defprocess(self,element,*args,**kwargs):try:team_info:TeamInfo=elementifteam_info.country=='France':yieldelementexceptExceptionaserr:failure=Failure(pipeline_step="Filter France teams",input_element=element,exception=err)yieldpvalue.TaggedOutput(FAILURES,failure)# In Beam pipeline.input_teams:PCollection[str]=p|'Read'>>beam.Create(team_names)outputs_map1,failures_map1=(input_teams|'Map to team with country'>>ParDo(MapToTeamWithCountry()).with_outputs(FAILURES,main='outputs'))outputs_map2,failures_map2=(outputs_map1|'Map to team with city'>>ParDo(MapToTeamWithCity()).with_outputs(FAILURES,main='outputs'))outputs_filter,failures_filter=(outputs_map2|'Filter France teams'>>ParDo(FilterFranceTeams()).with_outputs(FAILURES,main='outputs'))all_failures=(failures_map1,failures_map2,failures_filter)|'All Failures PCollections'>>beam.Flatten()This example starts with an inputPCollectioncontaining team names.Then 3 operations and steps are applied : 2 maps and 1 filter.For each operation a customDoFnclass is proposed and must overrideprocessfunction containing the
transformation logic.Atry except blocis added to catch all the possible errors.In theExceptbloc aFailureobject is built with input element and current exception. This object is then added on
atuple tagdedicated to errors.Thistagmechanism allows having multi sink in the pipeline and a dead letter queue for failures.There are some inconveniences :We have to repeat many technical codes and same logic liketry except bloc,tuple tags,failure logicand all this logic can be centralized.If we want to intercept all the possible errors in the pipeline, we have to repeat the recovery of output and failure in each step.All the failuresPCollectionmust be concatenated at end.The code is verbose.The repetition of technical codes is error-prone and less maintainable.Example of error handling using Asgarde library# Beam pipeline with Asgarde library.input_teams:PCollection[str]=p|'Read'>>beam.Create(team_names)result=(CollectionComposer.of(input_teams).map('Map with country',lambdatname:TeamInfo(name=tname,country=team_countries[tname],city='')).map('Map with city',lambdatinfo:TeamInfo(name=tinfo.name,country=tinfo.country,city=team_cities[tinfo.name])).filter('Filter french team',lambdatinfo:tinfo.country=='France'))result_outputs:PCollection[TeamInfo]=result.outputsresult_failures:PCollection[Failure]=result.failuresCollectionComposer classAsgarde proposes aCollectionComposerwrapper class instantiated from aPCollection.Operators exposed by CollectionComposer classTheCollectionComposerclass exposes the following operators :map,flatMapandfilter.These classical operators takes a function, the implementation can be :Alambda expressionAmethodhaving the same signature of the expectedfunctionFailure object exposed by AsgardeBehind the scene, for each step theCollectionComposerclass addstry exceptbloc andtuple tag logicwith output
and failuresinks.The bad sink is based on aFailureobject proposed by the library :@dataclassclassFailure:pipeline_step:strinput_element:strexception:ExceptionThis object contains the current pipeline step name, input element with string form and current exception.Input element on Failure object are built following these rules :If the current element in thePCollectionis adict, the Json string form of thisdictis retrievedFor all others types, thestringform of object is retrieved. If developers want to bring their own serialization
logic, they have to override__str__method in the object, example for adataclass:importdataclassesimportjsonfromdataclassesimportdataclass@dataclassclassTeam:name:strdef__str__(self)->str:returnjson.dumps(dataclasses.asdict(self))Result of CollectionComposer flowTheCollectionComposerclass after applying and chaining different operations, returns atuplewith :OutputPCollectionFailuresPCollectionresult=(CollectionComposer.of(input_teams).map('Map with country',lambdatname:TeamInfo(name=tname,country=team_countries[tname],city='')).map('Map with city',lambdatinfo:TeamInfo(name=tinfo.name,country=tinfo.country,city=team_cities[tinfo.name])).filter('Filter french team',lambdatinfo:tinfo.country=='France'))result_outputs:PCollection[TeamInfo]=result.outputsresult_failures:PCollection[Failure]=result.failuresExample of a flow with side inputsAsgardeallows applying transformations with error handling and passingside inputs.The syntax is the same as usual Beam pipeline withAsDictorAsListpassed as function parameters.defto_team_with_city(self,team_name:str,team_countries:Dict[str,str])->TeamInfo:returnTeamInfo(name=team_name,country=team_countries[team_name],city='')team_countries={'PSG':'France','OL':'France','Real':'Spain','ManU':'England'}# Side inputs.countries_side_inputs=p|'Countries'>>beam.Create(team_countries)# Beam Pipeline.result=(CollectionComposer.of(input_teams).map('Map with country',self.to_team_with_city,team_countries=AsDict(countries_side_inputs)).map('Map with city',lambdati:TeamInfo(name=ti.name,country=ti.country,city=team_cities[ti.name])).filter('Filter french team',lambdati:ti.country=='France'))result_outputs:PCollection[str]=result.outputsresult_failures:PCollection[Failure]=result.failuresAsgarde and error handling with Beam DoFn lifecyleAsgardeallows interacting withDoFnlifecycle while chaining transformation with error handling, example :(CollectionComposer.of(input_teams).map('Map to Team info',input_element_mapper=lambdateam_name:TeamInfo(name=team_name,country='test',city='test'),setup_action=lambda:print('Setup action'),start_bundle_action=lambda:print('Start bundle action'),finish_bundle_action=lambda:print('Finish bundle action'),teardown_action=lambda:print('Teardown action')))Themapandflat_mapmethods ofCollectionComposerclass propose the following functions to interact withDoFnlifecycle :setup_actionstart_bundle_actionfinish_bundle_actionteardown_actionThese functions take afunctionwithout input parameter and returnNone, it corresponds to an action executed
in the dedicated lifecycle method :https://beam.apache.org/documentation/transforms/python/elementwise/pardo/Advantage of using AsgardeAsgardepresents the following advantages :Simplifies error handling with less code and more expressive and concise codeNo need to repeat same technical code for error handling liketry exceptbloc,tuple tagsand concatenation of all the pipeline failuresAllows interacting with Beam lifecycle while chaining the transformation and error handling