Package: dataflows-normalize
Dataflows Resource write to db normalizedThis library provides some dataflows processing for normalizing a resource.It has special support for storing normalized data into DB tables.What is normalization?In short, it is the process of reducing duplication in a dataset.More can be read about this concepthere.ExampleLet's take, as an example, this world cities dataset (we shall call it thefactresource):fromdataflowsimportFlow,load,printerFlow(load('https://datahub.io/core/world-cities/r/world-cities.csv',name='cities'),printer(num_rows=1)).process()cities:#namecountrysubcountrygeonameid1les EscaldesAndorraEscaldes-Engordany30400512Andorra la VellaAndorraAndorra la Vella3041563...23018ChitungwizaZimbabweHarare1106542It seems that thecountryandsubcountrycolumns are quite repetitive - let's extract them into a separate, deduplicated resource (we will call that adimensionresource).To do that we use thenormalizeprocessor.This processor receives a single resource name, and a list ofNormGroupinstances. Each of these groups specifies one newdimensionresource to be extracted and deduplicated.Let's see it in action:fromdataflows_normalizeimportnormalize,NormGroupFlow(load('https://datahub.io/core/world-cities/r/world-cities.csv',name='cities'),normalize([NormGroup(['country','subcountry'],'country_id','id')],resource='cities'),printer()).process()cities:#namegeonameidcountry_id1les Escaldes304005102Andorra la Vella304156313Umm al Qaywayn29059424Ras al-Khaimah29107435Khawr Fakkān2916964...23014Bulawayo894701267723015Bindura895061267823016Beitbridge895269267923017Epworth1085510267623018Chitungwiza11065422676cities_country_id:#idcountrysubcountry130AfghanistanBadakhshan227AfghanistanBadghis321AfghanistanBalkh433AfghanistanBāmīān531AfghanistanFarah619AfghanistanFaryab728AfghanistanGhaznī813AfghanistanGhowr922AfghanistanHelmand1011AfghanistanHerat...26712677ZimbabweBulawayo26722676ZimbabweHarare26732673ZimbabweManicaland26742678ZimbabweMashonaland Central26752675ZimbabweMashonaland East26762674ZimbabweMashonaland West26772670ZimbabweMasvingo26782671ZimbabweMatabeleland North26792679ZimbabweMatabeleland South26802672ZimbabweMidlandsIf we follow the last line in the dataset (Chitungwiza), we can see that an entry for its region (Zimbabwe/Harare) was created with id2676, and that id was added to the original row instead of the original values.How much did we gain?The original CSV file has a size of 895,586 bytes.If we save the two new resources as CSVs, we would get542,299 bytes for thefactresource and 68,023 for the regionsdimensionresource - a total of 610,322 bytes (or a reduction of 31% in size).Not only this helps with size, it also improves greatly DB performance to store data in normalized form.DB NormalizationRunning similar code to above, only usingnormalize_to_dbwill do the following:Load existing values from databasedimensiontables (in case these tables exist)Normalize the input data, and split intofactanddimensionresourcesUpdate the DB tables with new values, while reusing existing referencesThe main difference in usage fromnormalizeis that the names of DB tables are provided.fromdataflows_normalizeimportnormalize_to_db,NormGroupFlow(load('https://datahub.io/core/world-cities/r/world-cities.csv',name='cities'),normalize_to_db([NormGroup(['country','subcountry'],'country_id','id',db_table='countries_db_table')],'cities_db_table','cities',db_connection_str='...'),).process()