Package: datawelder
Perform SQL-likeJOINson large file-like dataframes.Do you have tons of larger-than-memory datasets lying around on your file system?Do you dream of an easy way to join them together?Do you want to achieve this without using a database?If the answers to the above questions are “yes”, thendatawelderis for you!ExampleFirst, examine our toy dataset.
It contains country names and currencies in two separate tables.$head-n5sampledata/names.csviso3,nameAND,PrincipalityofAndorraARE,UnitedArabEmiratesAFG,IslamicRepublicofAfghanistanATG,AntiguaandBarbuda$head-n5sampledata/currencies.csviso3,currencyAND,EuroARE,DirhamAFG,AfghaniATG,DollarWe can join these two dataframes as follows:$python-mdatawelder.partitionsampledata/names.csvpartitions/names5$python-mdatawelder.partitionsampledata/currencies.csvpartitions/currencies5$python-mdatawelder.joinout.csvpartitions/namespartitions/currencies--formatcsv$grepANDout.csvAND,PrincipalityofAndorra,AND,EuroTweakingYou can specify the partition key explicitly:$python-mdatawelder.partitionsampledata/names.csvpartitions/names5--keyindex0$python-mdatawelder.partitionsampledata/names.csvpartitions/names5--keynameiso3You can specify any format parameters (e.g. CSV delimiter) explicitly:$python-mdatawelder.partitionsampledata/names.csvpartitions/names5--fmtparamsdelimiter=','lineterminator='\n'Similarly, for output:$python-mdatawelder.joinout.csvpartitions/namespartitions/currencies--formatcsv--fmtparamsdelimiter=;$grepANDout.csvAND;PrincipalityofAndorra;AND;EuroOther formats work transparently:$python-mdatawelder.partitionsampledata/names.jsonpartitions/names5--keynameiso3Once you partition your datasets, it doesn’t matter what format they were originally in.
You can merge them with any other partitioned dataset with ease:$python-mdatawelder.joinout.jsonpartitions/namespartitions/currencies--formatjson--subs1$head-n5out.json{"iso3":"AGO","name":"Republic of Angola","iso3_1":"AGO","currency":"Kwanza"}{"iso3":"AND","name":"Principality of Andorra","iso3_1":"AND","currency":"Euro"}{"iso3":"ARM","name":"Republic of Armenia","iso3_1":"ARM","currency":"Dram"}{"iso3":"ATF","name":"French Southern and Antarctic Lands","iso3_1":"ATF","currency":"Euro"}{"iso3":"AZE","name":"Republic of Azerbaijan","iso3_1":"AZE","currency":"Manat"}You can also select a subset of fields to keep (similar to SQL SELECT):$python-mdatawelder.joinout.csvpartitions/namespartitions/currencies--formatcsv--selectname,currency--subs1$head-n5out.csvname,currencyRepublicofAngola,KwanzaPrincipalityofAndorra,EuroRepublicofArmenia,DramFrenchSouthernandAntarcticLands,EuroThe name of each column is prefixed by the number of the dataframe it came from.
For example,1.currencymeans “the currency field from dataframe 1”.You can also rename the selected fields as desired (again, similar to SQL SELECT):$python-mdatawelder.joinout.csvpartitions/namespartitions/currencies--formatcsv--select'name as country_name, currency as curr'--subs1$head-n5out.csvcountry_name,currRepublicofAngola,KwanzaPrincipalityofAndorra,EuroRepublicofArmenia,DramFrenchSouthernandAntarcticLands,EuroFinally, you can use multiple processes for joining.
The default is the number of CPUs.
The order of the rows in the output file may differ due to race conditions,
but this does not affect the integrity of the data.$python-mdatawelder.joinout.csvpartitions/namespartitions/currencies--formatcsv--select'0.name as name, 1.currency as curr'--subs4How does it work?First,datawelderpartitionseach dataset using a partition key.
We used 5 partitions because the datasets are tiny, but you can specify an arbitrary partition size when working with real data.In this case, it automatically identified the format of the file as CSV.
You can give it a helping hand by specifying the format and relevant parameters (e.g. field separator, quoting, etc) manually.We did not specify a partition key to use in the above example, sodatawelderpicked a default for us (you can override this).
In the above example, we split each dataset into 5 partititions using the default key (whatever is the first column), but you can override that.FeaturesParallelization across multiple cores via subprocess/multiprocessingAccess to cloud storage for reading and writing e.g. S3 viasmart_open.  You do not have to store anything locally.Read/write various file formats (CSV, JSON, pickle) out of the boxFlexible API for dealing with file format edge cases