Package: pandas-mapper
pandas-mapperThe pandas-mapper is a Python pacckage that provides a concise syntax
for applying mapping transformations toPandasDataframes commonly required for
ETL workflows.  Possibly the biggest benefit to using pandas-mapper
over the nativepandas.Dataframe.applymethod is a robust error handling mechanism.  Instead of raising an
error, mapping errors can be redirected to an errors Dataframe, which
can then be handled by the user as needed.Getting startedTo get started, install pandas-mapper in your project using pippip install pandas-mapperand then use it your project by importing the packageimport pandas_mapperWhen you import this package in your project, it adds themappingmethod to Pandas dataframe
objects.  Suppose you had a dataframe containing integers and the English word for the integer
and you want to translate the names to Spanish.importpandasaspdimportpandas_mapperdf=pd.DataFrame({'num':[1,2,3],'name':['one','two','three'],'num_name':['1-one','2-two','3-three']})numnamenum_name1one1-one2two2-two3three3-threeA stupidly-simple translation method might bedeftranslate(val):ifval==1:return'uno'elifval==2:return'dos'elifval==3:return'tres'else:raiseValueError('Unknown translation:{}'.format(val))The translation can be accomplished using pandas-mapper viamapper=df.mapping([('num','translated',translate)])translated_df=mapper.mappedThe first argument of the mapping method is a list of tuples, where
the first element of the tuple is the source column(s), the second element
is the target column(s), and the (optional) third element is the
transform.  In this example, we only have one map in the list, so the result is
a dataframe with a single column:translatedunodostresHandling errorsOur stupidly-simple translation will raise an error if we supply it with the number4.  Suppose
we added another record with the number4to ourdfdefined above.  If we apply the same
mapping as above, pandas-mapper will raise aValueError.  However, if we supply themappingmethod with theon_error='redirect'option viamapper = df.mapping([('num', 'translated', translate)], on_error='redirect')
translated_df = mapper.mapped
translation_errors_df = mapper.errorsthen we get two dataframes, one with the translated records (mapper.mapped):translatedunodostresand another with the error records (mapper.errors):numnamenum_nameerrror4four4-four{'msg': 'ValueError(4): Unknown translation: 4...Mapping cardinalitiesEach map can be defined with 0 or more sources, 0 or more targets, and
0 or 1 transform functions.  The expected arguments and return values of the
transform function is depenedent on the number of source and target columns used.If the mapping has a single source and single target columns, then the
transform function should accept a single value and return a single value.If the mapping involves multiple source columns, then the
function should accept a single dict-like object where the
keys are the names of the source columns.  In this case, if the mapping
has a single target, then the retun value of the transform function should contain
a single value.  However, if the mapping has multiple targets, then the return
value should be the same dict-like object that was passed to the function, with
the target-column keys of that object have been modified in place by the function.If the mapping has no source columns, then the transform can either be a constant
(e.g., the integer 5), or a function that accepts no arguments but returns a value
(which may be useful if you want to use a random number generator).ExamplesZero-to-oneZero-to-one mappings can be used to set a column to a constant:df.mapping([None,'five',transform=5])Or defined by some function that generates output:importrandomdf.mapping([(None,'rando',random.random)]).mappedOne-to-oneOur translation function defined above is an example of a one-to-one transform:df.mapping([('num','translated',translate)])Many-to-oneConcatenation is an example of a many-to-one operation:df.mapping([(['num','name'],'num-name',lambdarow:'-'.join(row.apply(str)))])Many-to-ManyDeconcatenation is an example of a one-to-many operation.  One-to-many operations
require the same method signature as many-to-many:defdeconcatenate(row):split_values=row['num_name'].split('-')row[num'] = split_values[0]row[name'] = split_values[1]returnrowdf.mapping([('num_name',['num','name'],deconcatenate)])Other optionsThe mapping method also supports aninplaceoption, which isFalseby default.  This
will modify the dataframe in place, bringing along all columns that it started with.  For example:df.mapping([('num','translated',translate)],inplace=True).mappednumnamenum_nametranslated1one1-oneuno2two2-twodos3three3-threetresModifying the dataframe inplace can be useful when you need to chain together transformations,
like when the output of one map in needed as the input for another map.Contributor SetupDownload and install thedocker community editionthat is appropriate for your host os.We useinvoketo setup and control the environment
for developing and testing this project.  This will require that you install
invoke in your host OS.  You may be able to get away with just runningpip install invoke.  However, the recommended method is to download and installminiconda.  Then, create a project-specific
environment and install invoke in this environment:conda create --name pandas-mapper python=3.6
conda activate pandas-mapper
pip install invokeNote: if you use miniconda, you will have to runconda activate pandas-mappereach time you start a new terminal session.Once invoke is installed, you can build the docker containers to use the dev/test environmentinv buildSpin up the dev/test environment viainv up --jupyter-port=8892Run the test suite viainv test