Package: clevertable
CleverTableConsistent, intelligent transformation of text-based tabular data into numerical data.Minimal configuration required.Installation:pipinstallclevertableExample:fromclevertableimport*profile=ConversionProfile({# optionally specify converters for specific columns:"Country":OneHot(),"Diagnosis":Binary(positive="cancer",negative="benign"),"Hospitalized":None,# ignore column},pre_processing=None)df=profile.fit_transform("datasets/survey.xlsx")# transformed pandas.DataFrameWhy this Library?CleverTable makes it really easy to convert text-based tabular data
(optionally mixed with numbers) into numerical data, e.g. a medical survey
into a Pandas DataFrame or a NumPy array.If something is obvious, you should not need to specify it.
CleverTable will try to make choices for you if you don't make them.You stay in control: All choices made by CleverTable can be modified and overridden.This is how CleverTable works: (see below for a fulltutorial)You create a newprofile = ConversionProfile().
Here, you can optionally specify certain converters.You callprofile.fit(data)on a sample data set, which creates a fixed conversion profile.CleverTable chooses the best converter for each column if you don't specify it.The converter (chosen by you or by CleverTable) adapts its internal state to fit the data.You callprofile.transform(data)on the actual data set (which may be the same as forfit()),
which converts the data according to the fixed profile.Here are some examples on what you can do with CleverTable:Chain multiple converters to achieve complex conversions:profile["Column 7"]=[Split(),ForEach(Strip()),Flatten(),Infer()# Infer() -> CleverTable will choose what to put here]Use theInfer()converter where you want CleverTable to figure out the best solution (see above).Concise shorthand writings with Python syntax:profile["Column 1"]=[# Python lists create pipelinesstr.lower,# functions /lambdas:s.strip(),# lambda expressions are allowed]profile["Column 2"]={"Hello":1,"Bye":2}profile["Column 3"]=Float(),1# tries conversion to float, defaults to 1 on errorIncremental configuration: If a column already has a correct converter, you can further process the column
by adding another converter.
This implicitly creates a pipeline.profile["Column 5"]+=OneHot()Afterfit(), you can access the inferred state of the converters.my_weather_conv=profile["Weather"]# e.g. OneHot()my_weather_categories=my_weather_conv.values# e.g. ["sunny", "cloudy", "rainy"]Send multiple columns into one converter:profile["Column 1","Column 2"]=maxSend nested columns into one converter:profile[("Column A","Column B"),"Column C"]=[Parallel(max,floor),min]# min(max(A, B), floor(C))TutorialSuppose you want to convert the following table of survey results in a 2D numpy array of numbers:CountryAgeDiagnosisHospitalizedEducation levelSymptomsChina32benignnoUniversitycough, feverFrance45canceryesPhDfeverItaly19benignyesHigh SchoolcoughGermany56canceryesHigh Schoolfever and coughNigeria23benignnoUniversitycoughIndia34benignyesUniversitycough, fever..................For example, you might want to convert theCountrycolumn into a column of integers,
with every integer representing a different country.However:You don't really care which number represents which country.But you want to make sure that the same country always gets the same number,
even if you add more data to the table later.You also want to know which integer was chosen for which country.That's what CleverTable is for:First, you callfit()on a sample data set, which creates a fixed conversion profile.Then, you calltransform()on the actual data set, and it converts the data according
to the fixed profile.Moreover, CleverTable does many things automatically:It chooses the best converter if you don't specify it.And then, the converter also adapts its internal state to fit the data.Let's see how that works:fromclevertableimport*table="datasets/survey.xlsx"# filename or pandas.DataFrameprofile=ConversionProfile()profile.fit(table)# chooses best converters and creates a fixed conversion profileprint(profile)will show the inferred conversion profile:{"Country":Enumerate('china','france','germany',...),# lots of countries"Age":Float(),"Diagnosis":Binary(),"Hospitalized":Binary(),"Education level":OneHot('high school','phd','university'),"Symptoms":ListAndOr(),}We can access the individual converters and their properties by indexing the profile with the column name:country_converter=profile["Country"]# Enumerate('china', 'france', 'germany', ...)# see which integer corresponds to which country:countries_list=country_converter.values# ('china', 'france', 'germany', ...)You can now use this profile to convert data:# transform the whole table:df=profile.transform(table)# pandas.DataFramearr=df.to_numpy()# 2D numpy array# transform a single data point:data_point={"Country":"Germany"}transformed=profile.transform_single(data_point)# {'Country': 2}The nice thing is that you can now use the fixed profile
to find out after conversion where the numerical values originated from:# find out which country corresponds to the number 2:country_id=2country=profile["Country"].values[country_id]# 'germany'You may have noticed that all the strings appear in lowercase.
That is because theConversionProfilepre-processes all strings to lowercase by default.
You can disable this behavior by passingpre_processing=Noneto the constructor
or setting this property after construction:profile.pre_processing=None# disable pre-processingprofile.pre_processing=str.lower# default behaviorprofile.pre_processing=lambdas:s.strip().lower()It's okay to provide a pre-processing function that doesn't work for some entries
(e.g.str.lowerwill fail for non-string entries),
because CleverTable will catch errors and ignore them during pre-processing.You may also have noticed that theEducation levelcolumn was converted toOneHot(),
even though it contains arbitrary words, just like theCountrycolumn.
That's because CleverTable detected that there are too many different values
in theCountrycolumn for aOneHot()converter, so it chose theEnumerate()converter.But you can always override this behavior by explicitly setting the conversion method
before callingfit():fromclevertableimport*table="datasets/survey.xlsx"profile=ConversionProfile()# explicitly specify some converters:profile["Country"]=OneHot()profile["Diagnosis"]=Binary(positive="cancer",negative="benign")profile.fit(table)In this example, we also made sure that the "Diagnosis" column
is choosing the correct positive and negative values.You can also achieve the same by passing a dictionary to the constructor:fromclevertableimport*table="datasets/survey.xlsx"profile=ConversionProfile({"Country":OneHot(),"Diagnosis":Binary(positive="cancer",negative="benign"),}).fit(table)# fit() returns selfTwo final notes:You can ignore columns by setting their converter toNone(which is shorthand for theIgnore()converter).You can usefit_transform()to performfit()andtransform()with the same data in one call.This leaves us with this very concise code:fromclevertableimport*df=ConversionProfile({"Country":OneHot(),"Diagnosis":Binary(positive="cancer",negative="benign"),"Hospitalized":None,},pre_processing=None).fit_transform("datasets/survey.xlsx")Which produces the following transformed table:Country=ChinaCountry=France...Country=ZimbabweAgeDiagnosisEducation level=High SchoolEducation level=PhDEducation level=UniversitySymptoms=coughSymptoms=fever10...03200011101...04510100100...01901001000...05611001100...02300011000...034000111CLIpip install clevertablealso makes the commandclevertableavailable
in the command line.
It can convert files with tabular data.
Executeclevertable --helpto see what arguments can be passed to the tool:usage: clevertable [-h] [-i IGNORE [IGNORE ...]] src out

Consistent and intelligent conversion of tabular data into numerical values.

positional arguments:
  src                   Path to input file.
  out                   Path to output file.

optional arguments:
  -h, --help            show this help message and exit
  -i IGNORE [IGNORE ...], --ignore IGNORE [IGNORE ...]
                        Column names to ignore.How to ContributeBasic workflow of contribution:Fork the repositoryCreate a new branchMake your changesCreate a pull requestWait for the pull request to be accepted or rejectedIf accepted, you can delete your branchIf rejected, make the requested changes and push them to your branchRepeat until pull request is acceptedWhat to contribute:New converters (classes that inherit fromConverter)Improvements to converter inference (logic inInfer()converter)Improvements to default preprocessingMake more features available through the CLINew testsNew documentation, tutorials, examplesNew ideas, suggestions, bug reports → create an issue or contact me directlyDocumentationThere are only two classes that:ConversionProfile: A collection of converters.Converter: Transforms columns of data into columns of data.ConvertersHere's a quick overview of all converters:ConvertersDescriptionShorthandExample UsageBasic:Float()Convert numbers into floats.Enumerate()OneHot()Binary()Convert to 0 and 1. Detects common "positive" and "negative" terms in strings.List()ListAndOr()Map()dict{"foo": 1,"bar": -2,}Const()Return a constant value.any42"foo"Text Processing:Strip()Split()Combining Converters:Pipeline()Apply multiple converters in sequence.list[Split(),ForEach(Strip()),]Try()Try multiple converters and return the first one that succeeds.tuple(Float(), Binary())ForEach()Apply the same converter to all items.Parallel()Apply different converters to the respective items.Special:Id()Ignore()Drop the column.NoneNoneInfer()Label()Dimensionality:Flatten()Flatten a tuple of tuples into a single tuple. This is often needed afterForEach()orParallel().Transpose()Arbitrary Functions:Function()Apply a user-defined function to the data.callablelambda x: x**2StrictFunction()Apply a user-defined function to the data. Less flexible thanFunction().FloatConverts a column of numbers into a column of numbers.
If invalid values are encountered (NaN,inf,None, etc.),
a warning is printed and the value is replaced withnp.nan.
This can be circumvented by passing a value to thedefaultargument:"Temperature":Float(default=37.0)You can also specify"mean","median", or"mode"as the default value.
This will choose the default value based on the data in the specified column:"Temperature":Float(default="mean")Temperature⇒Temperature37.537.540.040.038.538.538.7539.039.0Results in:"Temperature":Float(default=38.75)EnumerateThis is the extension of theBinary()conversion method
to columns with more than two possible values.
The values are converted into integers starting at 0,
resulting in a single column of integers.The possible values can be passed to the constructor:"Country":Enumerate("france","germany","italy")Country⇒Countryfrance0italy2germany1Their index in the argument list is used as the numerical value.
If no values are specified, the values found in the provided
data are sorted in lexically ascending order.OneHotIf each entry contains one of multiple possible values.
The possible values can be specified via thevaluesargument:"Education Level":OneHot("primary","secondary","tertiary")Education Level⇒Education Level=primaryEducation Level=secondaryEducation Level=tertiaryprimary100secondary010tertiary001If no values are specified, the possible values are inferred from the data.BinarySimilar toEnumerate(), but with just two possible values,
and with some extra intelligence for this purpose.
For example, it can detect words commonly used for positive and negative values:Positive:yes,true,positive,1,femaleNegative:no,false,negative,0,male,noneExample:"Hospitalized":Binary()Hospitalized⇒Hospitalizedno0yes1false0true1none0Results in:"Hospitalized":Binary(positive={"yes","true"},negative={"no","false","none"})You can explicitly specify the values of thepositiveclass and thenegativeclass via the constructor:"Hospitalized":Binary(positive="yes",negative="no")Hospitalized⇒Hospitalizedyes1no0no0yes1If only one argument is specified (eitherpositiveornegative),
all other values present in the data are treated as instances of the other class:"Time served":Binary(negative="none")Time served⇒Time servednone01 year14 years1none0It's also possible to specify more than one value for thepositiveandnegativeclasses.
Example:"Hospitalized":Binary(positive={"yes","true"},negative={"no","false"})Hospitalized⇒Hospitalizedyes1no0false0true1If no positive or negative values are specified, a set of strings commonly used
to indicate positive / negative values is tested against the available data.
For instance, in the example above, the specified arguments would have been
inferred automatically as positive and negative.If this approach is not successful, the lexically smallest value is chosen as thenegativeargument and
thepositiveargument is left empty, causing all other values to be treated as positive:"Fruits":Binary()Fruits⇒Fruitsbanana1apple0kiwi1apple0Results in:"Fruits":Binary(negative="apple")ListConverts lists of values into multiple binary columns."Symptoms":List()Symptoms⇒Symptoms=coughSymptoms=feverSymptoms=headachefever, cough, headache111headache, cough101The default delimiter is a comma.You can specify a custom delimiter via thedelimiterargument:"Symptoms":List(delimiter=";")"Symptoms":List(delimiter=[",",";"])# also accepts listsThe passed strings are interpreted as regular expressions.ListAndOr"Symptoms":ListAndOr()Symptoms⇒Symptoms=coughSymptoms=feverSymptoms=headachefever, cough and headache111headache or cough101The default delimiters are comma, "and" and "or".The passed strings are interpreted as regular expressions.MapStripSplitPipelineTryTry(converter1,converter2,...)Returns value of the first converter that does not raise an exception,
or the original value if all converters raise an exception.Try()always only applies one converter and returns its output (if it didn't fail)."Product":Try(Float(),Infer())# will infer the converter for the samples that cannot be converted to floatsProduct⇒ProductKiwi48Apple0712356712356261382261382Banana1Kiwi48......This would result in the following profile afterfit():"Product":Try(Float(),Enumerate("Apple","Banana",...))ForEachApply the same converter to all items.ParallelParallel(converter1,converter2,...)Apply different converters to the respective items.
Usually used in a Pipeline after other converters that create outputs with multiple items (e.g.Split()).
Also, you usually want to useFlatten()after this, as each individual converter will return a tuple of items,
even if it only contains one item.
Example:"Latitude;Longitude":[Split(";"),# must always result in two items, because Parallel() has 2 convertersParallel(Ignore(),Float()),# ignore latitude, convert longitude to float -> [[], [longitude]]Flatten(),# -> [longitude]]Latitude;Longitude⇒Longitude52.520008;13.40495413.40495448.137154;11.57612411.576124ConstIdIdentity.
Keeps the input unchanged.IgnoreDrops the column."registration_timestamp":NoneThis is chosen if no appropriate conversion method could be found.InferTries to infer the conversion method from the column name.
Afterfit(), this converter will be replaced with the inferred converter in the profile.This is the default converter for columns where no converter is specified.
This converter can however also be used anywhere else explicitly.
Examples:"col1":[str.upper,Infer()],"col2":Try(Float(),Infer()),# will infer the converter for the samples that cannot be converted to floatsLabelFlattenTransposeCan transpose nested tuples, given that the nested tuples are of equal length.For example, look at this elegant implementation of theList()converter:"Symptoms":[Split(r"\s*,\s*"),# split at commaForEach(OneHot()),Transpose(),ForEach(max),Flatten()]Transpose()allows us to applymaxto each column of the one-hot encodings
across all tuple elements.FunctionFunction(transform,labels=None)Shorthand: Instead ofFunction(transform, None), just writetransform, wheretransformis some callable.Creates a custom converter from a customtransform()function
(and optionally, a customlabels()function).
This is a handy way to create a converter that doesn't needfit().UnlikeStrictFunction(), this class can handle functions that don't accept or return tuples,
which often allows for more concise code.This is achieved duringfit()as follows:If all incoming items are 1-element tuples, it sets a flagUNPACK_OUTPUTto always
unpack the element before passing them to the wrapped function duringtransform().If duringfit()the wrapped function doesn't return tuples,
it tries to turn that output into a tuple:If the output is always a non-string iterable, it will simply set a
flagCONVERT_ITERABLEto always convert the iterable output into a tuple duringtransform().Otherwise, it sets a flagWRAP_OUTPUTto always wrap the output in a
1-element tuple duringtransform().A similar logic is applied to the labels.
If a custom labels function is given, the following procedure is followed duringlabels():If the incoming labels are a 1-element tuples, the single label is unpacked before
it is passed to the custom labels function.If the custom labels function returns something other than a tuple,
this class tries to convert it into a tuple:If the output is a non-string iterable, it is converted into a tuple.Otherwise, the output is wrapped in a 1-element tuple.If no custom labels function is given, the output labels are generated based on the
output cardinality inferred duringfit()and according to the following logic:If the number of incoming labels is identical to the output cardinality,
the labels will be returned unchanged.If there are multiple incoming labels but a single output label,
the output label is formed by joining the incoming labels with,.If there is a single incoming label but multiple output labels,
the output labels are formed by adding suffixes_0,_1, etc.
to the single input label.1 Output LabelM Output Labels1 Input Labelidenticalsuffixes_0,_1, ...N Input Labelsjoin with,identical if M=N, else ERRORA special case are functions returning output of varying cardinality duringfit().
In this case, a single label is returned.
If only one input label is given, that single label is returned.
If multiple input labels are given, they are joined with_.The following example turns a text column into two columns containing the ascii code of the first and last letter."Name":lambdax:(ord(x[0]),ord(x[-1]))Name⇒Name_0Name_1Alice97101Bob9898(Remember that by default, all text entries are converted to
lowercase before further processing.)As you can see, the number of columns is inferred directly from the return value of the conversion function.
If the function returns a tuple, the resulting column names are indexed.You can also set the labels explicitly with a lambda function
that takes the input column name as an argument and returns output column names:"Name":Function(lambdax:(ord(x[0]),ord(x[-1])),labels=lambdas:(f"ord(first letter of{s})",f"ord(last letter of{s})")),Name⇒ord(first letter of Name)ord(last letter of Name)Alice97101Bob9898However, remember that you can always simply useLabel()to rename the columns after the conversion,
if you don't need the output column names to depend on the input column names."Name":[lambdax:(ord(x[0]),ord(x[-1])),Labels("ord(first letter)","ord(last letter)")],StrictFunctionStrictFunction(transform,labels=None)Works mostly likeFunction(), but simpler:transformandlabelsmust both accept and return tuples.
Instead of something like this:"Name":str.lower,you have to write this:"Name":StrictFunction(lambdax:(str.lower(x[0]),))# notice the comma, which makes it a 1-element tupleThat is, you will still receive 1-element tuples as tuples to the function,
even if all input elements duringfit()are 1-element tuples.
Also, you must now explicitly return a tuple,
even if it is just a 1-element tuple, as otherwise an error will be raised.SeeFunction()for a convenient extension of this converter.Understanding Multi-Column ConvertersA converter returns two things:transform(): the items of the transformed datalabels(): a label for each itemBoth return values are tuples.For top-level converters, this then creates the corresponding amount of columns.
This includes the case ofa 1-element tuple(item,), which is the case for most converters.an empty tuple(), in which the result is ignored.
(In fact, this is exactly howIgnore()is implemented.)This means, however, that for top-level converters,labels()andtransform()must return the same number of items.
That is becauselabels()is used to create the output column names.
Iftransform()returns a different number of items, that will raise an error for top-level converters.However, for nested converters,labels()andtransform()can return different numbers of items.
For exampleSplit.labels()always returns only one item,
because the number of items returned bySplit.transform()varies from input to input.
Therefore,Split()can't be used as a top-level converter
and has to be used inside aPipelineor similar devices,
so that other converters can ensure that the final output is of constant size.