Package: uhura
UhuraA framework for tackling legacy data engineering code.Wrap inputs and outputs for your system and take advantage of asyncio + automated testing.Uhura makes it easy to run pipelines in a specific "mode". Modes are implemented using context managers and are stored in themodesmodule. Right now Uhura supports two main use cases:A testing mode used to run a pipeline and compare its output against pre-generated fixtures. This is executed through thefixture_builder_modeand thetask_test_modecontexts.A mode used for testing whether functions respect specific properties. This is executed through thetest_transformerscontext.More details on how to run the two modes can be found in the Examples section belowExamples1. Pipeline testing modeImagine you have some legacy ETL code:importpandasaspddefload_data()->pd.DataFrame:...# IO code heredefload_ids_of_interest()->set[int]:...# More IO codeasyncdefwrite_filtered_data(df:pd.DataFrame):...asyncdefmain():data=load_data()ids=load_ids_of_interest()awaitwrite_filtered_data(data[data['id'].isin(ids)])These functions might be reading from, and writing to, live cloud resources. They may also be loadinglarge amounts of data, all of which makes it tricky/ unsafe to run this code locally.If something goes wrong we can use logging to try and understand what's going on in the cloud. Butprint statement debugging is no substitute for a proper local debugger. If we had decently separatedIO and business logic we might be able to use local unit tests, but the existing code might not beset up for this, and refactoring without tests in the first place is quite risky.To solve this problem, uhura allows you to wrap arbritary IO code, and substitute it when testing.It also contains tools to speed up the process of building local end-to-end tests.So what does this wrapping process look like?First we decorate our functions which provide inputs or outputs:importpandasaspdfromuhura.functionalimportuhura_reader,uhura_writer@uhura_readerdefload_data()->pd.DataFrame:...# IO code here@uhura_readerdefload_ids_of_interest()->set[int]:...# More IO code@uhura_writerasyncdefwrite_filtered_data(df:pd.DataFrame):...Uhura supports both async and sync input and output functions. As well as generators and async iterators.But what do we gain from doing this? The answer lies in the uhura 'modes'.Modes let us change the behaviour of our uhura classes to gain certain benefits. A goodstarting example is the 'fixture_builder_mode' which lets us store a local copy of eachinput and output, which can speed up testing when our underlying code is reaching out theinternet / running queries, and also means that we don't actually write anything out to thereal systems we interact with.For example:fromuhura.modesimportfixture_builder_modefromtimeimportsleep@uhura_readerdefexpensive_input():sleep(10)# Expensive IO operationreturn1withfixture_builder_mode():assertexpensive_input()==1# This takes a while...assertexpensive_input()==1# This executes instantly!# Now we can start testing. First we need to stage our data.fromuhura.modesimportfixture_builder_modefromasyncioimportrunwithfixture_builder_mode():run(main())By running our code in this context manager, we replace each reader with a 'pull through proxy' whichwill call the underlying code and save the result to a location in the local filesystem. It will onlydo this is there isn't already some data there, which lets us quickly rebuild parts of our fixtureswithout having to rebuild everything.Now it's time to test things.fromuhura.modesimporttask_test_modewithtask_test_mode():main()In this mode, all inputs will be replaced with the staged inputs saved by the fixture_builder_mode,and all outputs will be automatically compared with the staged outputs. If the outputs differ, Uhurawill raise an exception highlighting exactly what's changed.Class based APIDecorating functions will address common requirements, but uhura also supports having reader and writer classes.These are often useful for when a resource is both read and written within a codebase.fromuhura.baseimportReadable,Writablefromuhura.modesimportfixture_builder_modeclassItemClient(Readable[int],Writable[int]):defread(self):return1defwrite(self,obj):print(f"Writing{obj}")passwithfixture_builder_mode():client=ItemClient()client.write(client.read()+1)# Caches locallyThe class based API also has some additional features beyond the basic decorators, such as overriding the cachekey (see below).GotchasCaching MechanismNote, the readers and writersdo nottake the arguments to their constructors into account, there will be onefixture for each reader and writer by default._fake_database={"id1":27,"id2":34}classDatabaseReader(Readable[int]):def__init__(self,id_:str):self._id=id_defread(self):return_fake_database[self._id]withfixture_builder_mode():assertDatabaseReader("id1").read()==27assertDatabaseReader("id2").read()==34# This will fail it returns just 27 againIf the required arguments are known at definition time you can create multiple classes or functions for differentresources. Alternately this behaviour can be overridden by implementing the 'cache_key' hook method on acorresponding class._fake_database={"id1":27,"id2":34}classDatabaseReader(Readable[int]):def__init__(self,id_:str):self._id=id_defread(self):return_fake_database[self._id]defcache_key(self):returnf"DatabaseReader/{self._id}"withfixture_builder_mode():assertDatabaseReader("id1").read()==27assertDatabaseReader("id2").read()==34# Success!InheritanceUhura will only apply changes to classes which directly subclass Uhura types, not classes subclassed from those initialclasses.Subclasses of those classes will only immediately work with Uhura if they delegate all the relevant methods to theirparent (e.g. read, write, cache_key).You can safely inherit from Uhura base types at multiple points in your hierarchy if needed.Custom ComparisonsIn task_test_mode, the default comparison is normally fairly good. But in some situations it won'tbe sufficient. This includes when outputs include things like 'last_processed' timestamps. Thefollowing example gives an instance of this:fromdatetimeimportdatetimeclassOutputWriter(Writable[dict]):defwrite(self,obj:dict):...def write_out_result(result: int):return OutputWriter().write({"result": result, "time_processed": datetime.now()})What we can do in this case is add a pre-processor to the comparison for the relevant class. Thesyntax for that is as follows.fromuhura.modesimportdefault_comparatordef_drop_time_processed(output:dict):deloutput["time_processed"]returnoutputdefault_comparator["OutputWriter"].add_preprocessor(_drop_time_processed)withtask_test_mode():...Editing FixturesFixtures are stored as pickle files, by default under:known_good_path = "tests/fixtures/output_known_good"input_path = "tests/fixtures/input"The known good path storing written outputs.There is one file for each client, named '{client_name}.pkl'.If you have specified a custom cache_key, this will form the file name of the fixture instead.These can be edited manually (for instance if you want to downsample data). Relevant outputscan then be deleted and regenerated using the fixture_builder_mode.Custom SerdeBy default uhura stores all fixtures as pickle files, in some settings other formats may bepreferable. This can be specified by overriding the 'get_serde' method on readers and writers.importpandasaspdfromuhura.serdeimportParquetSerdeclassParquetReader(Readable[pd.DataFrame]):defread(self):return_load_from_external()defget_serde(self):returnParquetSerde(read_kwargs={"engine":"fastparquet"})Additional serde classes can be implemented by implementing the 'uhura.serde.Serde' interface.2. Properties testing modeLet's say that you want to know whether a function operating on a dataframe will return a different output if the dataframe is sorted differently or whether a function will modify a specific column in the dataframe. Rather than reading the functions code you can write a property to check for these behaviors and then let Uhura figure out whether these functions behave as expected.Properties are stored in theuhura/properties/testable_propertiesmodule. As an example let's look at the property checking whether a function output is affected by the order of records in the dataframe passed as an argumentimportpandasaspdfromtypingimportOptionalfromuhura.properties.homomorphic_hashimporthomomorphic_hashfromuhura.properties.testable_propertiesimportSENTINELdeforder_invariant(tested,arg:pd.DataFrame,result:Optional[pd.DataFrame]=None):"""Check whether `tested` output is affected by the order of the rows in `arg`Args:tested (Callable): the function for which we are testing the propertyarg (pd.DataFrame): the dataframe over which we want to run the function being testedresult (pd.DataFrame): the result of `tested(arg)` this may already be passed as an argument in case it isavailable, and we don't need to run again `tested(arg)` to check the property. So, that we can avoidrunning `tested` unnecessarily.Returns:pd.DataFrame: The output of running the tested function on the main dataframe ie `tested(arg)`bool: It tells if the property is respected"""shuffled_result=homomorphic_hash(tested(arg.sample(frac=1)).sort_index())result=tested(arg)ifresultisSENTINELelseresultreturnresult,homomorphic_hash(result.sort_index())==shuffled_resultAll properties should have the same signature, as per docstring. Optionally a property can also return a third object, a string containing some notes regarding the property.In order to run a property on a specific function you need to decorate the function with the transformer method.importpandasaspdfromuhura.propertiesimporttransformer@transformerdeffunction_that_respect_property(df:pd.DataFrame):...returndf@transformer(evaluated_arg_name="df_b")deffunction_that_respect_property(df_a:pd.DataFrame,df_b:pd.DataFrame):...returndf_ain case the function takes multiple arguments you can specify which one should be used with the function when testing the property. In order to do so it's enough to pass its name toevaluated_arg_namein the transformer.Most of the time the property is tested on the function by comparing the input and the main output of the function. So, the function has to have a single output for the workflow to work as expected. That output is what is going to be compared against the function input.Once you have decorated the functions you care about property-checking you need to specify what properties you want to test while running the pipeline. You can easily do so through thePropertyTesterobject. Let's say I want to test whether a function output is going to be affected by how its input dataframe is sorted and also whether a few specific columns are being modified (col_aandcol_d)fromuhura.propertiesimportPropertyTester,order_invariant,get_columns_not_modified_propertytester=PropertyTester.create_for_properties([order_invariant,get_columns_not_modified_property(["col_a","col_d"])])Once we defined the property functions, we decorated the functions we want to test the properties on, and we created a property_tester object we can then use this to run the property testing mode.fromuhura.modesimporttest_transformersfromuhura.propertiesimportPropertyTester,order_invariant,get_columns_not_modified_propertydefmain():# our main pipeline including the functions we decorated with `@transformer`...passtester=PropertyTester.create_for_properties([order_invariant,get_columns_not_modified_property(["col_a","col_d"])])withtest_transformers(tester):main()By default, the logs are used to show whether the properties are respected on the functions. However, the output can be redirected somewhere else if preferred. As an example you can find below how the properties info appears in the logs.2023-05-02 10:48:58,580 [INFO] uhura: [find_maiden_voyages]: has property order_invariant  
2023-05-02 10:48:58,610 [INFO] uhura: [find_maiden_voyages]: has property get_columns_not_modified_property.<locals>.columns_not_modified -  
2023-05-02 10:48:58,749 [INFO] uhura: [prepare_vessel_status]: lacks property order_invariant  
2023-05-02 10:48:59,810 [INFO] uhura: [insert_voyage_id]: lacks property order_invariant  
2023-05-02 10:49:01,423 [INFO] uhura: [prepare_model_output]: lacks property order_invariantCaveatsWhen importing a library in another codebase by default its logger will be disabled, so if you want the uhura logs to be made available you will need to explicitly enable it, as per below:importloggingfromuhura.propertiesimportUHURA_PROPERTIES_LOGGER_NAMElogging.getLogger(UHURA_PROPERTIES_LOGGER_NAME).disabled=False