Numeric Encoding Options with Automunge

1 INTRODUCTION . Of the various modalities of machine learning application ( e.g . images , language , audio , etc . ) tabular data , aka structured data , as may comprise tables of feature set columns and collected sample rows , in my experience does not command as much attention from the research community , for which I speculate may be partly attributed to the general non-uniformity across manifestations precluding the conventions of most other modalities for representative benchmarks and availability of pre-trained architectures as could be adapted with fine-tuning to practical applications . That is not to say that tabular data lacks points of uniformity across data sets , for at its core the various feature sets can at a high level be grouped into just two primary types : numeric and categoric . It was the focus of a recent paper by this author ( Author , 2020 ) to explore methods of preparing categoric sets for machine learning as are available in the Automunge open source python library platform for tabular data pipelines . This paper will give similar treatment for methods to prepare numeric feature sets for machine learning . Of course it would be an oversimplification to characterize “ numeric feature sets ” as a sufficient descriptor alone to represent the wide amount of diversity as may be found between different such instances . Numeric could be referring to integers , floats , or combinations thereof . The set of entries could be bounded , the potential range of entries could be bounded on the left , right , or both sides , the distribution of values could be thin or fat tailed , single or multi-modal . The order of samples could be independent or sequential . In some cases the values could themselves be an encoded representation of a categoric feature . Beyond the potential diversity found within our numeric features , another source of diversity could be considered based on relationships between multiple feature sets . For example one feature could be independent of the others , could contain full or partial redundancy with one or more other variables by correlation , or in the case of sequential data there could even be causal relationships between variables across time steps . The primary focus of transformations to be discussed in this paper will not take into account variable interdependencies , and will instead operate under the assumption that the training operation of a downstream learning algorithm may be more suitable for the efficient interpretation of such interdependencies , as the convention for Automunge is that data transformations ( and in some cases sets of transformations ) are to be directed for application to a distinct feature set as input . In many cases the basis for these transformations will be properties derived from the received feature in a designated “ train ” set ( as may be passed to the automunge ( . ) function ) for subsequent application on a consistent basis to a designated “ test ” set ( as may be passed to the postmunge ( . ) function ) . 2 NORMALIZATIONS . A common practice for preprocessing numeric feature sets for the application of neural networks is to apply a normalization operation in which received values are centered and scaled based on properties of the data . By conversion to comparable scale between features , backpropagation may have an easier time navigating the fitness landscape rather than overweighting to higher magnitude inputs ( Ng , 2011 ) . Table 1 surveys a few normalization operations as available in Automunge . Upon inspection a few points of differentiation become evident . The choice of denominator can be material to the result , for while both ( max - min ) and standard deviation can have the result of shrinking or enlarging the values to fall within a more uniform range , the ( max - min ) variety has more of a known returned range for the output that is independent of the feature set distribution properties , thus allowing us to ensure all of the min-max returned values are non-negative for instance , as may be a pre-requisite for some kinds of algorithms . Of course this known range of output relies on the assumption that the range of values in subsequent test sets will correspond to the train set properties that serve as a basis - to allow a user to prevent these type of outliers from interfering with downstream applications , Automunge allows a user to pass parameters to the transformation functions , such as to activate floors or caps on the returned range . An easy to overlook outcome of the shifting and/or centering of the returned range by way of the subtraction operation in the numerator is a loss of the original zero point , as for example with z-score normalization the returned zero point is shifted to coincide with the original mean . It is the opinion of this author that such re-centering of the data may not always be a trivial trade-off . Consider the special properties of the number 0 in mathematics , including multiplicative properties at/above/below . By shifting the original zero point we are presenting a ( small ) obstacle to the training operation in order to relearn this point . Perhaps more importantly , further trade-offs include the interpretability of the returned data . Automunge thus offers a novel form of normalization , available in our library as ‘ retn ’ ( standing for “ retain ” ) , that bases the formula applied to scale data on the range of values found within the train set , with the result of scaling the data within a similar range as some of those demonstrated above while also retaining the zero point and thus the +/- sign of all received data . 3 TRANSFORMATIONS . In many cases the application of a normalization procedure may be preceded by one or more types of data transformations applied to the received numeric set . Examples of data transformations could include basic mathematic operators like + - * / , log transforms , raising to a power , absolute values , etc . In some cases the transformations may also be tailored to the properties of the train set , for example with a Box-Cox power law transformation ( Box & Cox , 1964 ) . In the Automunge library , the order of such sets of transformations , as may be applied to a distinct source column , and in some cases which may include generations and branches of derivations , are specified by way of transformation category entries to a set of “ family tree ” primitives [ Table 3 ] ( Author , 2020 ) for a root transformation category , and where a transformation category entry may be associated with one or more transformation functions intended for application to corresponding train and/or test set feature columns , potentially including custom transformation functions which may be defined with minimal requirements of simple data structures . Such root categories may be pre-defined in the Automunge library of transformations or may be custom configured by a user in entries to a “ transformdict ” data structure . The convention for transformation functions in the Automunge library is that any kind of function accepts any kind of data , and in cases where an invalid entry is returned , for example when dividing by zero or taking a square root of a negative number , such entry may serve as a target for missing data infill , with infill methods that may be applied to a column from a library of infill options - including “ ML infill ” in which random forest models ( Breiman , 2001 ) are used to predict infill based on properties of the train set . To facilitate the application of infill , transformation categories used as root categories are specified with a classification for the types of data that will be considered valid input , as for example may be non-negative numeric , non-zero numeric , integer numeric , etc . These “ NArowtype ” classifications are populated in the same “ processdict ” data structure used to assign transformation functions to a transformation category . 4 BINS AND GRAININGS . In most cases the transformations considered in the preceding section maintained full information retention of the received data , such that with returned sets the form of the input data can be recovered with an inversion operation ( as is available in the Automunge library ) . For binning transformations , there may instead be a type of coarse graining of the feature set to aggregate buckets of entries into a categoric representation . Automunge offers a wide range of options for numeric binning [ Table 4 ] . Bins may be aggregated to either supplement or replace received numeric sets . For each binning operation , transformation category options are available to return the categoric encoding as a one-hot encoding , ordinal integer encoding , or binary encoding in which distinct categories may be represented by multiple simultaneous activations . This is partly motivated by different conventions of various libraries for accepting input to an entity embedding layer ( Guo & Berkhahn , 2016 ) as may be applied to the returned categoric encoding in a downstream training operation . 5 NOISE INJECTION . For most cases in the Automunge library , transformations applied to a train set feature set are applied to the corresponding test set feature set using the same basis , such that if the same data is received for both train and test sets , the same form will be returned ( a useful point for validations ) . The noise injection options are a little different in that such injections may be intended just for the train data but not the corresponding test data . The rationale behind noise injections were first to support differential privacy considerations ( Dwork et al. , 2006 ) . Other potential uses of noise injections could be to perturb the model training , facilitating diversity between models as may be beneficial in the aggregation of ensembles ( Dietterich , 2000 ) or as a source of training data augmentation ( Perez & Wang , 2017 ) . The experiments detailed below will suggest a material model performance benefit from data augmentation by noise injection in cases of underserved training data , which we believe is a novel innovation for tabular learning . The options available for noise injection are generally derived by way of aggregations of transformation category entries to family tree primitives , as noise injection may be applied downstream to a normalization or categoric encoding . ( The convention for transformation functions is that they receive input of a single target column , so transformations performed downstream of a categoric encoding should be fed an ordinal input . ) The library includes distinct noise injection family tree aggregations tailored to operation of several different types of received normalizations , as may rely on a known range or scale of input , or applied preceding different types of categoric encodings . Numeric noise injections [ Table 5 , Fig 1 ] are derived from a Gaussian source with configurable parameters . For noise intended to sets with a fixed range of values such as DPmm , although the noise source as implemented is Gaussian , the application is capped from extreme outliers at half of range ( e.g . +/- 0.5 ) and based on whether an input entry is above or below the midpoint , positive or negative noise respectively is scaled to ensure maintained original range in returned data based on values of input entry . Parameters are also accepted to indicate what ratio of input will receive injection . Similar options are available for Laplace distribution noise profiles . To clarify how family tree primitives come into play , for the ‘ DPmm ’ root category in Fig 1 , when ‘ DPmm ’ is applied as a root category to a source column with header ‘ column ’ , the upstream primitive entries are inspected , which include a parents entry of the ‘ DPm2 ’ transformation category and a cousins entry of the ‘ NArw ’ transformation category . For the ‘ NArw ’ upstream entry to the cousins primitive , the ‘ NArw ’ transformation category is associated with a NArw transformation function ( a function aggregating boolean activations indicating presence of infill ) per the processdict entry associated with the ‘ NArw ’ root category , which transformation function returns the column ‘ column NArw ’ with the suffix appender ‘ NArw ’ logging the transformation function applied . Because cousins is a primitive without offspring no further generations are inspected in the ‘ NArw ’ family tree downstream primitives . For the upstream parents primitive entry ‘ DPm2 ’ , the ‘ DPm2 ’ processdict entry has a transformation function of mnmx ( min-max scaling ) , which is applied and logged by the ‘ mnmx ’ suffix appender , and since parents is a primitive with offspring the downstream primitives in the ‘ DPm2 ’ family tree are inspected where a coworkers entry of ‘ DPmm ’ transformation category is found which is associated with a DPmm transformation function ( noise injection corresponding to range 0-1 ) based on the ‘ DPmm ’ root category processdict entry . Because coworkers is a replacement primitive the column configuration with header ‘ column mnmx ’ is not retained for the returned set . The returned column originating from the DPmm transformation function has header logging the two applied transformation functions as ‘ column mnmx DPmm ’ . Please note that column retention , as may be impacted by the application of downstream replacement primitive entries , is signaled in these diagrams by the color coding between orange and blue , and applied transformation functions are shaded as green . Transformation category entries of primitives that are not inspected are left without shading . Also please note that the processdict entries here are an abstraction for the set of corresponding transformation functions that may be directed to train and/or test set feature sets . The application of noise to categoric encodings [ Table 6 , Fig 2 ] is a little simpler , where a given ratio of entries in a categoric feature set are flipped to one of the other activations , between which have a uniform probability of replacement ( including possibility of original entry retention ) . For the ‘ DP10 ’ root category example shown in Fig 2 , this is achieved by first applying an ordinal encoding by ord3 transformation function associated with a ‘ DPo6 ’ transformation category entry to the parents upstream primitive of the ‘ DP10 ’ root category , then a noise injection by the DPod transformation function associated with a ‘ DPo3 ’ transformation category to the downstream children primitive associated with the ‘ DPo6 ’ root category ( applied since parents is a primitive with offspring ) , followed by a binary encoding by the 1010 transformation function associated with the ‘ 1010 ’ transformation category entry to the downstream coworkers primitive associated with the ‘ DPo3 ’ root category ( applied since children is a primitive with offspring ) . Here the intermediate stages of derivations associated with columns ‘ column ord3 ’ and ‘ column ord3 DPod ’ are not retained in the returned set since children and coworkers are replacement primitives .

Summary: This paper introduces a number of data preprocessing options for numeric features provided by an open source library automunge. The most of the paper focuses on explaining the specific transformations offered under each option, including normalization, binning, and noise injection. For normalization, a new transformation 'retain' is offered in addition to traditional z-score, min-max etc. The paper uses one section between normalization and binning to explain a notion 'family tree primitives' which is used in the composition of multiple transformations. In experiments, the paper uses the higgs dataset. Three settings of the dataset are used: full data, 5% data and 0.25% data. 6 settings of transformations are used: raw data, z-score, retain, retain with bins, retain with noise injection, and retain with partial noise injection. When averaged over the three settings of the dataset, using raw data leads to suboptimal auc score compared with the other five. It is acknowledged that "the metrics for normalized data were slightly better than raw on average, it is not clear if they were sufficiently statistically significant to draw firm conclusions. " The paper concludes that "any consideration around benefits of feature engineering should distinguish