Package: sk-transformers
sk-transformersA collection of various pandas & scikit-learn compatible transformers for all kinds of preprocessing and feature engineering steps🛠IntroductionEvery tabular data is different. Every column needs to be treated differently. Pandas is already great! Andscikit-learnhas a nicecollection of dataset transformers. But the possibilities of data transformation are infinite. This project tries to provide a brought collection of data transformers that can be easily used together withscikit-learn- either in apipelineor just on its own. See theusage chapterfor some examples.The idea is simple. It is like a well-equipped toolbox 🧰: You always find the tool you need and sometimes you get inspired by seeing a tool you did not know before. Please feel free tocontributeyour tools and ideas.Check out some examples in theJupyter notebook.InstallationIf you are usingpip, you can install the package with the following command:pipinstallsk-transformersIf you are usingPoetry, you can install the package with the following command:poetryaddsk-transformersinstalling dependenciesWithpip:pipinstall-rrequirements.txtWithPoetry:poetryinstallAvailable transformersModuleTransformerDescriptionDatetime transformerDateColumnsTransformerSplits a date column into multiple columns.Datetime transformerDurationCalculatorTransformerCalculates the duration between to given dates.Encoder transformerMeanEncoderTransformerScikit-learn API for thefeature-engine MeanEncoder.Generic transformerAggregateTransformerThis transformer uses Pandas groupby method and aggregate to apply function on a column grouped by another column.Generic transformerAllowedValuesTransformerThis transformer replaces values that arenotin a list with another value.Generic transformerColumnDropperTransformerDrops columns from a dataframe using Pandas drop method.Generic transformerColumnEvalTransformerProvides the possibility to use Pandas methods on columns.Generic transformerDtypeTransformerTransformer that converts a column to a different dtype.Generic transformerFunctionsTransformerThis transformer is a plain wrapper around thesklearn.preprocessing.FunctionTransformer.Generic transformerLeftJoinTransformerUses Pandas merge function to perform a left-join based on the column of a dataframe and the index of another dataframe. The right dataframe is essentially a lookup table.Generic transformerMapTransformerThis transformer iterates over all columns in thefeatureslist and applies the given callback to the column. For this it uses thepandas.Series.mapmethod.Generic transformerNaNTransformerReplace NaN values with a specified value. Internally Pandas fillna method is used.Generic transformerQueryTransformerApplies a list of queries to a dataframe. If it operates on a dataset used for supervised learning this transformer should be applied on the dataframe containingXandy.Generic transformerValueIndicatorTransformerAdds a column to a dataframe indicating if a value is equal to a specified value.Generic transformerValueReplacerTransformerUses Pandas replace method to replace values in a column.Number transformerMathExpressionTransformerApplies an operation to a column and a given value or column. The operation can be any operation from thenumpyoroperatorpackage.Number transformerGeoDistanceTransformerCalculates the distance in kilometers between two places on the earth using the latitudes and longitudes.String transformerEmailTransformerTransforms an email address into multiple features.String transformerIPAddressEncoderTransformerEncodes IPv4 and IPv6 strings addresses to a float representation.String transformerPhoneTransformerTransforms a phone number into multiple features.String transformerStringSimilarityTransformerCalculates the similarity between two strings using thegestalt pattern matchingalgorithm from theSequenceMatcherclass.String transformerStringSlicerTransformerSlices all entries of specified string features using the slice() function.String transformerStringSplitterTransformerSplits a string column into multiple columns based on the occurrence of a character.String transformerStringCombinationTransformerContatenates two string columns after ordering them alphabetically first.UsageLet's assume you want to use some method from [NumPy's mathematical functions, to sum up the values of columnfooand columnbar. You could
use theMathExpressionTransformer.importpandasaspdfromsk_transformersimportMathExpressionTransformerX=pd.DataFrame({"foo":[1,2,3],"bar":[4,5,6]})transformer=MathExpressionTransformer([("foo","np.sum","bar",{"axis":0})])transformer.fit_transform(X).to_numpy()array([[1, 4, 5],
       [2, 5, 7],
       [3, 6, 9]])Even if we only pass one tuple to the transformer - in this example. Like with most other transformers the idea is to simplify preprocessing by giving the possibility to operate on multiple columns at the same time. In this case, theMathExpressionTransformerhas created an extra column with the namefoo_sum_bar.In the next example, we additionally add theMapTransformer.
Together withscikit-learn's pipelinesit would look like this:importpandasaspdfromsk_transformersimportMathExpressionTransformerfromsk_transformersimportMapTransformerfromsklearn.pipelineimportPipelineX=pd.DataFrame({"foo":[1,2,3],"bar":[4,5,6]})map_step=MapTransformer([("foo",lambdax:x+100)])sum_step=MathExpressionTransformer([("foo","np.sum","bar",{"axis":0})])pipeline=Pipeline([("map_step",map_step),("sum_step",sum_step)])pipeline.fit_transform(X)foo  bar  foo_sum_bar
0  101    4          105
1  102    5          107
2  103    6          109ContributingWe're all kind of in the same boat. Preprocessing/feature engineering in data science is somehow very individual - every feature is different and must be handled and processed differently. But somehow we all have the same problems: sometimes date columns have to be changed. Sometimes strings have to be formatted, sometimes durations have to be calculated, etc. There is a huge number of preprocessing possibilities but we all use the same tools.scikit-learns pipelineshelp to use formalized functions. So why not also share these so-called transformers with others? This open-source project has the goal to collect useful preprocessing pipeline steps. Let us all collect what we used for preprocessing and share it with others. This way we can all benefit from each other's work and save a lot of time. So if you have a preprocessing step that you use regularly, please feel free to contribute it to this project. The idea is that this is not only a toolbox but also an inspiration for what is possible. Maybe you have not thought about this preprocessing step before.Please check out theguideon how to contribute to this project.