Numeric Encoding Options with Automunge
Authors: Nicholas Teague
Venue:  (2021)
Keywords: tabular
feature engineering
preprocessing

Abstract: Mainstream practice in machine learning with tabular data may take for granted that any feature engineering beyond scaling for numeric sets is superfluous in context of deep neural networks. This paper will offer arguments for potential benefits of extended encodings of numeric streams in deep learning by way of a survey of options for numeric transformations as available in the Automunge open source python library platform for tabular data pipelines, where transformations may be applied to distinct columns in “family tree” sets with generations and branches of derivations. Automunge transformation options include normalization, binning, noise injection, derivatives, and more. The aggregation of these methods into family tree sets of transformations are demonstrated for use to present numeric features to machine learning in multiple configurations of varying information content, as may be applied to encode numeric sets of unknown interpretation. Experiments demonstrate the realization of a novel generalized solution to data augmentation by noise injection for tabular learning, as may materially benefit model performance in applications with underserved training data.

Under review as a conference paper at ICLR 2021
NUMERIC ENCODING OPTIONS WITH AUTOMUNGE
Anonymous authors Paper under double-blind review
ABSTRACT
Mainstream practice in machine learning with tabular data may take for granted that any feature engineering beyond scaling for numeric sets is superfluous in con- text of deep neural networks. This paper will offer arguments for potential benefits of extended encodings of numeric streams deep learning by way of survey of options for numeric transformations as available in the Automunge open source python library platform for tabular data pipelines, where transformations may be applied to distinct columns in "family tree" sets with generations and branches of derivations. Automunge transformation options include normalization, binning, noise injection, derivatives, and more. The aggregation of these methods into family tree sets of transformations are demonstrated for use to present numeric features to machine learning in multiple configurations of varying information content, as may be applied to encode numeric sets of unknown interpretation. Experiments demonstrate the realization of novel generalized solution to data augmentation by noise injection for tabular learning, as may materially benefit model performance in applications with underserved training data.
INTRODUCTION
Of the various modalities of machine learning application (e.g. images, language, audio, etc.) tabular data, aka structured data, as may comprise tables of feature set columns and collected sample rows, in my experience does not command as much attention from the research community, for which I speculate may be partly attributed to the general non non-uniformity across manifestations precluding the conventions of most other modalities for representative benchmarks and availability of pre-trained architectures as could be adapted with fine-tuning to practical applications. That not to say that tabular data lacks points of uniformity across data sets, for at its core the various feature sets can at a high level be grouped into just two primary types: numeric and categoric. It was the focus of recent paper by this author (Author, 2020) explore methods of preparing categoric sets for machine learning as are available in the Automunge open source python library platform for tabular data pipelines. This paper will give similar treatment for methods to prepare numeric feature sets for machine learning.
Of course it would be oversimplification characterize "numeric feature sets" sufficient descriptor alone to represent the wide amount of diversity as may be found between different such instances. Numeric could be referring to integers, floats, or combinations thereof. The set of entries could be bounded, the potential range of entries could be bounded on the left, right, or both sides, the distribution of values could be thin or fat tailed, single multi-modal. The order of samples could be independent or sequential. In some cases the values could themselves be an encoded representation of categoric feature.
Beyond the potential diversity found within our numeric features, another source of diversity could be considered based on relationships between multiple feature sets. For example one feature could be independent of the others, could contain full partial redundancy with ables by correlation, or in the case of sequential data there could even be causal relationships be- tween variables across time steps.
more other vari-
The primary focus of transformations to be discussed in this paper will not take into account vari- able interdependencies, and will instead operate under the assumption that the training operation of a downstream learning algorithm may be more suitable for the efficient interpretation of such interdependencies, the convention for Automunge is that data transformations (and in some cases

---

Under review as a conference paper at ICLR 2021
sets of transformations) are to be directed for application distinct feature set input. In many cases the basis for these transformations will be properties derived from the received feature designated "train" set (as may be passed to the automunge()) function) for subsequent application on a consistent basis to designated "test" set (as may be passed to the postmunge(.) function).
2 NORMALIZATIONS
A common practice for preprocessing numeric feature sets for the application of neural networks to apply normalization operation in which received values are centered and scaled based on properties of the data. By conversion to comparable scale between features, backpropagation may have an easier time navigating the fitness landscape rather than overweighting higher magnitude inputs (Ng, 2011). Table I surveys few normalization operations as available in Automunge.
Table I: Normalizations
Type
ID
Formula
Scaling
z-score
'nmbr'
(x: pa/o
scaled to sigma and mu 0 scaled to unit interval
min-max
'mnmx'
(x; min)/ (amn)//mar-maae - min) min)
mean
'mean"
(r; meen)/////eemomememem /maaa min)
scaled and centered to mean scaled by median absolute deviation log-normal scaled to Gaussian
MAD
'MA3''
(x; - -max)//// (MLA)
lognorm
'lgmm'
In(x;) + (z. 14/0
Upon inspection a few points of differentiation become evident. The choice of denominator can be material the result, for while both (max min) and standard deviation have the result of shrinking or enlarging the values to fall within a more uniform range, the (max min) variety has more of known returned range for the output that is independent of the feature set distribution properties, thus allowing us to ensure all of the min-max returned values are non-negative for in- stance, as may be a pre-requisite for some kinds of algorithms. Of course this known range of output relies on the assumption that the range of values in subsequent test sets will correspond to the train set properties that serve as a basis to allow user to prevent these type of outliers from interfering with downstream applications, Automunge allows user to pass parameters to the transformation functions, such as to activate floors or caps on the returned range.
An easy overlook outcome of the shifting and/or centering of the returned range by way of the subtraction operation in the numerator is a loss of the original zero point, as for example with z-score normalization the returned zero point is shifted to coincide with the original mean. It is the opinion of this author that such re-centering of the data may not always be a trivial trade-off. Consider the special properties of the number ( in mathematics, including multiplicative properties atfabove/below. By shifting the original zero point we are presenting a (small) obstacle to the training operation in order relearn this point. Perhaps more importantly, further trade-off's include the interpretability of the returned data.
Automunge thus offers novel form of normalization, available in library as 'retn' (standing for "retain"), that bases the formula applied to scale data on the range of values found within the train set, with the result of scaling the data within similar range as some of those demonstrated above while also retaining the zero point and thus the +/- sign of all received data.
Table 2: Retain Normalization ('retn')
Min
Max
Formula
Returned Min
Returned Max
A VIA OOO
A VIV
Ti/ Ti/(max- - min)
min/ (maz min)
max (max min)
0
(I; - min)/(maz min) 0
0
(x; maa))(mar min)
-1
2

---

Under review as a conference paper at ICLR 2021
3 TRANSFORMATIONS
In many cases the application of a normalization procedure may be preceded by one or more types of data transformations applied the received numeric set. Examples of data transformations could include basic mathematic operators like + /, log transforms, raising to power, absolute values, etc. In some cases the transformations may also be tailored to the properties of the train set, for example with Box-Cox power law transformation (Box & Cox, 1964).
In the Automunge library, the order of such sets of transformations, as may applied distinct source column, and in some cases which may include generations and branches of derivations, are specified by way of transformation category entries (Author, 2020) for root transformation category, and where transformation category entry may be associated with one or more transformation functions intended for application to corresponding train and/or test set feature columns, potentially including custom transformation functions which may defined with minimal requirements of simple data structures. Such root categories may be pre-defined in the Automunge library of transformations may be custom configured by user in entries to a "transformdict" data structure.
of "family tree" primitives [Table 3]
Table 3: Family Tree Primitives
Upstream/
Applied to Generation
Downstream Offssring
Primitive
Column Action
Downstream
parents
upstream
first first first
replace
yes
siblings
upstream
supplement
yes
auntsuncles
upstream
no replace
cousins
upstream
first
supplement
no
children
downstream parents
offspring
replace
yes
niecesnephews
downstream siblings
offspring
supplement yes replace
coworkers
downstream auntsuncles
offspring offspring
friends
downstream cousins
supplement
no
The convention for transformation functions in the Automunge library is that any kind function accepts any kind of data, and in cases where invalid entry returned, for example when dividing by zero or taking square root of a negative number, such entry may serve as a target for missing data infill, with infill methods that may be applied to column from a library of infill options including "ML infill" in which random forest models (Breiman, 2001) are used to predict infill based on properties of the train set. To facilitate the application infill, transformation categories used as root categories are specified with a classification for the types of data that will be considered valid input, as for example may non-negative numeric, non-zero numeric, integer numeric, etc. These "NArowtype" classifications are populated in the same "processdict" data structure used to assign transformation functions to a transformation category.
BINS AND GRAININGS
In most cases the transformations considered the preceding section maintained full information retention of the received data, such that with returned sets the form of the input data can be recovered with an inversion operation (as is available in the Automunge library). For binning transformations, there may instead be type of coarse graining of the feature set to aggregate buckets of entries into categoric representation. Automunge offers wide range options for numeric binning [Table 4].
Bins may be aggregated either supplement replace received numeric sets.
For each binning operation, transformation category options are available to return the categoric encoding one-hot encoding, ordinal integer encoding, or binary encoding in which distinct categories may be represented by multiple simultaneous activations. This is partly motivated by different conventions of various libraries for accepting input to entity embedding layer (Guo & Berkhahn, 2016) as may be applied to the returned categoric encoding a downstream training operation.
3

---

Under review as a conference paper at ICLR 2021
Table 4: Binning Transformation Category Options
Transform
One-Hot Ordinal
Binary
Parameters
Number of standard deviations from the mean
'bins'
'bsor'
'bsbn'
'bincount'
Powers of ten
pwws'
'pwor'
'wwbn' 'po33'
Powers ten (with support for negative) Fixed width bins
pwr2'
'por2'
'bnwd'
'bnwo'
'bnwb'
"width"
Equal population bins User specified bins (first/last unconstrained)
'bnep'
'bneo'
"bneb"
"bincount"
'bkll'
'bkt3'
"bkb3'
'buckets'
User specified bins (first/last bounded)
'kk12'
'bk14"
"bkb4'
'buckets'
NOISE INJECTION
For most cases in the Automunge library, transformations applied to train set feature set are applied the corresponding test set feature using the same basis, such that if the same data received for both train and test sets, the same form will be returned (a useful point for validations). The noise injection options are little different that such injections may intended just for the train data but not the corresponding test data.
The rationale behind noise injections were first to support differential privacy considerations (Dwork et al., 2006). Other potential uses of noise injections could be perturb the model training, facil- itating diversity between models as may be beneficial the aggregation of ensembles (Dietterich, 2000)
source of training data augmentation (Perez & Wang, 2017). The experiments detailed below will suggest a material model performance benefit from data augmentation by noise injection in cases of underserved training data, which believe a novel innovation for tabular learning.
The options available for noise injection are generally derived by way of aggregations of transfor- mation category entries to family tree primitives, noise injection may be applied downstream to a normalization or categoric encoding. (The convention for transformation functions is that they receive input of a single target column, so transformations performed downstream of categoric encoding should be fed an ordinal input.) The library includes distinct noise injection family tree aggregations tailored operation of several different types of received normalizations, as may rely on known range scale of input, or applied preceding different types of categoric encodings.
Table 5: Numeric Noise Injections
Root Category
Normalization
Noise Type
Parameters
'DPnb" 'DPmm' 'DPrt'
'nmbr' "mmmx" 'retn"
Gaussian w/ Bernoulli ratio
'mu' / 'sigma' 'flip ciro'p
scaled Gaussian w/ Bernoulli ratio scaled Gaussian w/ Bernoulli ratio
"mu' /'iigma' / 'flip prob' "mu' /siimma' 'flip-prob'
Table 6: Categoric Noise Injections
Root Category
Encoding
Noise Type
Parameters
'DPbn'
'bnry' (boolcan)
Bernoulli flip
'llip "lip-prob'' 'llip prob' 'flip prob' 'flip-prob'
'DPod'
'ord3' (ordinal)
Bernoulli flip to random activation Bernuuli flip random activation '1010' (binary)
'DPoh'
'onht' (one-hot)
'DP10'
Bernoulli flip to random activation set
Numeric noise injections [Table 5, Fig 1] are derived from a Gaussian source with configurable parameters. For noise intended to sets with fixed range of values such DPmm, although the
4

---

Under review as a conference paper at ICLR 2021
root category
'DPmm!
DPm2
NVA
DPr
primiiive
perents
['DPmd''
(DPw2) ]
siblings
()
()
suntsunces
NNNA
[DPP
tranaformel
['NArw']
(NAR
(NNAWW
downaf prindives
chiidren
()
nessssssphss
[1
coworkers
(1
('Peen')
friands
(1
(I
procesedici
DPmm
mome
NAW
DPn
columm mmmx _ww_______mmmmmmm____ DPmm
(support entries for DPrr))
(support entries for DPmm)
coiumn, DPttt
returned columns
colurm NAo NAww
column
Figure 1: 'DPmm' and 'DPrt' family trees
root category
DP10
DPul'
DPod
'1010'
upatrream primiii
parents
[1DPe8]
[DPul'
siblings
()
transformddee
auntsuncles
[1000') (NNAWW]
cousirn
[NNW
NAIW
[NAww]
dowretrsem primiives
childion
()
(DP33)
[]
11
rlocesnephows
coworkers
(1010''
frends
[]
processdict
DPod
en33
DPad
1010
column ood3 DPodd 1010 (1010 returs set)
(support entries for DP10)
(support entries for DP10)
(support antries for DP10)
returned columns
Nolkmm_NAAw
Figure 2: 'DP10' family trees
noise source implemented is Gaussian, the application is capped from extreme outliers at half of range (e.g. +/- 0.5) and based on whether an input entry is above or below the midpoint, positive or negative noise respectively is scaled ensure maintained original range in returned data based on values of input entry. Parameters are also accepted to indicate what ratio of input will receive injection. Similar options are available for Laplace distribution noise profiles.
To clarify how family tree primitives come into play, for the 'DPmm' root category in Fig 1, when DDmmmm is applied as a root category to a source column with header 'column', the upstream primi- tive entries are inspected, which include parents entry of the 'DPm2' transformation category and a cousins entry of the 'NArw' transformation category. For the 'NArw' upstream entry to the cousins primitive, the 'NArw' transformation category is associated with a NArw transformation function (a function aggregating boolean activations indicating presence of infill) per the processdict entry associated with the 'NArw' root category, which transformation function returns the column 'col- NArw' with the suffix appender 'NArww' logging the transformation function applicd. Because cousins is a primitive without offspring further generations inspected in the 'NArw' family tree downstream primitives. For the upstream parents primitive entry 'DPm2', the 'DPm2' process- dict entry has a transformation function of mnmx (min-max scaling), which is applied and logged by the _mnmx' suffix appender, and since parents is primitive with offspring the downstream prim- itives in the 'DPm2' family tree are inspected where coworkers entry of 'DPmm' transformation category is found which is associated with a DPmm transformation function (noise injection corre- sponding to range 0-1) based on the 'DPmm' root category processdict entry. Because coworkers is replacement primitive the column configuration with header 'column mnmx' is not retained for the returned set. The returned column originating from the DPmm transformation function has header logging the two applied transformation functions as 'column_mman ______mmm_______________ mmmm'. Please note that column retention, as may be impacted by the application downstream replacement primitive entries, is signaled in these diagrams by the color coding between orange and blue, and applied transformation functions are shaded as green. Transformation category entries of primitives that not inspected are left without shading. Also please note that the processdict entries here an abstraction for the set of corresponding transformation functions that may be directed to train and/or test set feature sets.
5

---

Under review as a conference paper at ICLR 2021
The application of noise to categoric encodings [Table 6, Fig 2] is a little simpler, where given ratio of entries in a categorie feature set are flipped to one of the other activations, between which have a uniform probability of replacement (including possibility of original entry retention). For the 'DP10' root category example shown in Fig 2, this is achieved by first applying an ordinal encoding by ord3 transformation function associated with 'DPo6' transformation category entry the parents up- stream primitive of the 'DP10' root category, then a noise injection by the DPod transformation function associated with a 'DPo3' transformation category the downstream children primitive as- sociated with the 'DPo6' root category (applied since parents is primitive with offspring), followed by binary encoding by the 1010 transformation function associated with the '1010' transforma- tion category entry to the downstream coworkers primitive associated with the 'DPo3' root category (applied since children associated with columns "column_ord3 and Column_ord3_DPoa are not retained in the returned set since children and coworkers are replacement primitives.
primitive with offspring). Here the intermediate stages of derivations
SEQUENTIAL DATA
For numeric features in which the order of samples carry some significance, e.g. for time series data, some additional options are available to extract structure from relationships between time steps (these methods may benefit from a convention that time deltas between measurements are at or nearly constant). The theory that for sequential machine learning applications, as may make use of recurrence, convolution, or attention mechanisms, there may be benefit to supplementing feature streams with properties carried forward from prior time steps, where this type of operation may be particularly beneficial when there is some known cyclic property inherent in the application, e.g. for scheduled trading hours or quarterly reports.
Included in the Automunge library are sequential transforms to supplement sequential streams with proxies for derivatives by returning deltas between an entry and some desired time step prior by way of the 'dxdt' family of transforms [Fig 3]. Such application may be applied once as proxy for velocity, and may also be run multiple times upon that output as proxies for higher order derivatives. A variant on this operation may, instead of taking point-wise deltas, return deltas between averages of sets of points, which has the effect of smoothing or de-noising the data. The outputs of these operations may each be normalized such as with the 'retn' normalization for sign retention.
root category
'dedf"
upotream printivee
parents
['dxdr
'd2dt"
(dBdt'
[I
[]
transformdict
sunfounciee
["retrr')
('tebr')
['een']
cousins
[NAAww]
('NArw' ]
NNAww'
downatream primiives
children
[]
()
()
rlocesnephows
[]
[dxdtt)
(d2ott
cowcrker
("ratn')
('rotn")
]
fronnd
[]
[]
processdict
dxdi
dxdtt
dxdit
column dxdi rain column rein cokurn_NAr
column, dxdi dect
column, dedi, dedi, didt roin column, dedi, rein columin _ikei_reenn column, rein column Nolum_NAAww
returned columns
columnnreenn
column_NAAn NNAww
columin
Figure 3: 'dxdt' family trees
7 INTEGER SETS
Integer feature sets of unknown origin may present a particular challenge for automated encodings, as these may be associated with diverse set of interpretations, and may originate from continu- ous variables, counters, discrete relational variables (e.g. small/medium/large)) or possibly even an ordinal categoric encoding (Stevens al., 2020). The Automunge philosophy for these kind of am- biguities is to simply redundantly encode in a manner suitable for each [Table 7, Fig 4], defering to training operation for determining relevancy.

---

Under review as a conference paper at ICLR 2021
Table 7: Integer Encoding Options
Transform
Type
Useful For
'retn'
Retain normalization
Continuous variables
"pwr2"
Order of magnitude bins
Continuous variables
'ordl'
Ordinal
Ordered categoric
"1010"
Binary
Discrete categoric
'dxdt'
Sequential
Cumulative sequential
Frequency sorted ordinal followed by min-max
Scaled metric for ranking entry frequency
root catagory
hilg
vig?
'mnmoe
uppernsem primitves
parents
['igg']
['rgg2]
['rg2]
sblings
()
[]
transformdic
auntsuncles
Irretr, '010;, 1
I'retrr, 11010, 'orrr, 'pwrz
rettr, loref, ppor2]
couslins
[NNAn/
[NAw
['NAnw]]
('NArw')
downsdroa prebbres
chidton
[]
ricceeraphews
[]
cowerkerss
''mmmx')
'mreme'
I'mmmee
processdiet
and3
column_ord3 mann column
column_one Saann_ood____mmee mmmmx
column_sdd3 Souumn_srd__mmm grdd__mmmx
(ouport migr]
column,
column ratn
cokkmn_1010 (swl)
column_1010 (see)
column gnd
returned columns
column ord
colum, ordi
column por2
column__Nr NAww
column, par2 (sat)
column NA
column NAr
Figure 4: 'ntgr' family trees
EXPERIMENTS
Some experiments were run to evaluate impact of various normalizations and transformations. The Higgs data set (Baldi et al., 2014) was selected based on scale and predominantly numeric fea- ture types. The Higgs application is tabular data for binary classification sourced from high energy physics domain to evaluate subatomic particle interactions to distinguish between background noise and traces of Higgs boson interactions. The origin paper noted some details of architectures (5 layers with 300 nodes each) that served as basis for the experiments, and we used a 0.6M size validation set out of the 8.8M samples. A departure was made from the origin paper in use of phased learning rate with the fastai (Howard data, partly for convenience along with time and resource constraints. Since the interest primar- ily to evaluate relative performance between different preprocessing methods, no significant attempt was made to train models maximum performance to venture beyond double descent, instead the primary tuning aspect was selecting an epoch depth beyond which additional did not have improved validation metries. The experiments were repeated with different size subsets of the training data, including full data set, 5% samples, and 0.25% samples to represent scenarios with underserved training data. Learning rates deferred the fastai default finder, and the evaluation metric was lected as area under the receiver operating characteristic curve (ROC AUC) to be consistent with the origin demonstration.
Gugger, 2020) "fit_one_cycle" learner for tabular
The experiments applied Automunge to preprocess the feature sets, in each case applying uniform transformation types between features although with basis fit to properties of each respective col- umn, and with missing data infill by way of the Automunge adjacent cell infill option. The types of transformations were selected to demonstrate impact few representative variants noted this writeup, and included a base scenario with no feature scaling applied (just raw numeric data), along with a scenario for z-score normalization, retain normalization, retain normalization supplemented by standard deviation bins, two scenarios of retain normalization with noise injection (with noise injected to all entries or to 3% entries in each feature), and finally a data augmentation with re- tain normalized data supplemented by doubling the number of samples include retain with partial
7

---

Under review as a conference paper at ICLR 2021
Table 8: Higgs Data Normalization Scenarios (AUC metric // compared raw data)
Retain w/
Retain w/
Retain with Bins
Retain w/ Augment
Raw Data
Z-Score
Retain
Noise full
Noise partial
full data 42 epochs
0.8670
0,8668 (0.0002)
0.8667
0.8660 (0.0010)
0.8268 (0.0402)
0.8624 (0.0046)
0.8667
(0.0003)
(0.0003)
5% data 14 epochs
0.8436
0.8432
0.8431
0.8433
0.7679
0.8388
0.8453
(0.0004)
(0.0005)
(0.0003)
(0.0757)
(0.0048)
0.0017
0.25% data 3 epochs
0.7715
0.7716
0.7716
0.7718 0.0003
0,7052 (0.0663)
0,7668 (0.0047)
0.7821
0.0001
0.0001
0.0106
noise injection. Both of the noise injections applied Gaussian noise with standard deviation of 0.03 and with scaling to maintain the fixed range of the received data.
The findings of the experiments are summarized in Table 8 by way of reporting the final validation set metrics and performance delta from the raw data scenario. The AUC metric shown compares to the origin paper's 's reported 88%, which was based on a reported 200-1 ,,000 epochs compared to this experiment's 3-42 so there is some dissimilarity in results, which can also partly be attributed the phased learning rate schedule applied by fastai. The AUC metrics shown are averages of repetition counts detailed in Appendix G.
The scenarios for raw data, 7-score, and retain normalization achieved similar results. Part of the impact of supplementing the normalized data with standard deviation bins was small increase of training time. As expected the noise injection had dampening effect on the model accuracy, although the partial injection had only a small penalty in comparison the retain normalization without injection. Data augmentation improved the results, sufficiently to draw the conclusion of material benefit scenarios with underserved training data. We believe the realization of a novel generalized solution to data augmentation by noise injection for tabular learning is a significant finding.
The experiment with the full data set was repeated to demonstrate impact to linear model:
Table 9: Higgs Data Normalization Scenarios (Support Vector Classifier with Linear kernel)
Retain w/ Noise full
Retain w/
Retain w/ Augment partial
Retain with Bins
Raw Data
Z-Score
Retain
Noise partial
0.6410
0.6410
0.6410
0.6821 0.0411
0.6201 (0.0209)
0.6393 (0.0018)
0.6402
Accuracy
(0.0008)
DISCUSSION
I would offer first that any consideration around benefits feature engineering should distinguish first by scale of data available for training. When approaching big data scale input with infinite computational resources there may be less case to be made for much beyond basic normalized input. The nuance comes into play when are targeting applications with real world constraints. Although deep over- over-parama parameterized models may still be applied to target applications with under- represented data (Olson et al., 2018), such methods carry overheads. It is one of the premises of
8

---

Under review as a conference paper at ICLR 2021
the Automunge library that supplementing data streams with redundant features in multiple configurations may lower the bar to efficient extraction of inter-variable relationships.
My experience is that the machine learning community has by large become somewhat dismissive of any kind of feature engincering. expect partly owed to such works as Deep Learning (Goodfel- low al., 2016) which offers that deep learning has supplanted the need for such effort. I would offer in retort that there are different kinds of feature engineering to consider. Those methods as may apply explorations and optimizations between inter-variable relationships I agree are largely redundant of what may be efficiently derived through backpropagation and this is part of the reason why Automunge hasn't 't ventured into this territory. But the types of data manipulations used to supplement numeric features with multiple configurations are computationally efficient, and need not require sophisticated optimizations to apply. The primary cost of such supplements are the memory overheads as the data set size is expanded.
may be
Further, variations on feature composition, such as by variations information content and varia- tions on injected noise, may serve as useful source of model perturbations in ensemble assemblies. Noise injection may of benefit for anonymizing sensitive data for purposes of differential pri- vacy, and may also serve as a source of training data augmentation, similar to how for convolutional networks training data images may be duplicated with artificial variations (Perez & Wang, 2017).
I'll close by noting interesting paper I saw at NeurIPS last year, "SGD on Neural Networks Learn Functions of Increasing Complexity" by Nakkiran et al. (2019), in which the authors found that SGD behaves linear model in early epochs, and importantly that characteristics of such linear models are retained even in the later stages of training. Put differently, any steps that may be made to ensure that our models may efficiently extract properties even in early stages, before deep learning can do its magic, are not immaterial to the final performance.
Table 10: Summary of Use Cases
Desciption
Examples
Use Cases
Z-Score Normalization
nmbr
general use, when distribution is unknown
Min-Max Scaling
mamx
when fixed range is desired, e.g. nonnegative
Retain Normalization
retn
when interpretability is desired, sign retention
bins,
to aggregate sets into buckets, such as to sup- plement normalized data, helpful with linear models
Bins and Grainings
pwrs,
bnep
DPnb,
DPmm,
useful for data set augmentation, especially with underserved training data, also for differ- ential privacy
Noise Injection
DPrt, DPod,
DP10
dxdt, d2dt, d3dt
useful for time-series data, supplements nor- malized sets with proxies for derivatives
Sequential
ntgr, ntg2, to redundantly encode integer sets of unknown ntg3
Integer Sets
interpretation

---

Under review as a conference paper at ICLR 2021
ACKNOWLEDGMENTS
A thank you is owed to Alice Zheng and Amanda Casari whose 2018 book "Feature Engineering for Machine Learning" served as a helpful reference as I began to explore the practice of feature en- gineering. Thanks to Stack Overflow, Python, PyPI, GitHub, Colaboratory, Anaconda, and Jupyter. Special thanks to Scikit-Learn, Numpy, Scipy Stats, and Pandas.
REFERENCES
Author. String theory: Parsed categoric encodings with automunge, 2020.
Pierre Baldi, Peter Sadowski, and Daniel Whiteson. Searching for exotic particles in high- energy physics with deep learning. Nature Communications, 5(1):4308, 2014. doi: 10.1038/ ncomms5308. URL https: / /dii org/ /10. 1...1111111... 1038/nccomm5308.
George E. P. Box and D. Cox. An analysis of transformations. Journal of the Royal Statis- tical Society: Series B (Methodological), 26(2):211-243, 1964. doi: https://doi.org/10.1111
Leo Breiman. Random forests. Machine Learning, 45(1):5-32, 2001. doi: 10.1023/A:
1010933404324.
Thomas G. Dietterich. Ensemble methods in machine learning. MCS 2000. Lecture Notes in Computer Science, 1857, 2000.
Multiple Classifier Systems. doi: https://doi.org/10.100
3-540-45014-9.1
Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. TCC'06: Proceedings of the Third conference on Theory of Cryptography, 17:265-284, 2006. doi: /1tp://000..///1000000//1/61111
14.
lan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep Leurning. MIT Press, 2016.
Cheng Guo and Felix Berkhahn. Entity embeddings of categorical variables. 1604.06737, 2016.
ArXiv preprint
Jeremy Howard and Sylvian Gugger. Deep Learning for Coders with fastai and PyTorch. O' O'Reilly Media, 2020.
Preetum Nakkiran, Gal Kaplun, Dimitris Kalimeris, Tristan Yang, Benjamin L. Edelman, Fred Zhang, and Boaz Barak. Sgd on neural networks learns functions of increasing complexity. In NeurlPS, 2019.
Andrew Ng. Machine Learning, 2011. URL machi neeeiie--lle me--eearing,
Matthew Olson, Abraham Wyner, and Richard Berk. Modern neural networks generalize on small data sets. In NeurlPS, 2018.
Luis Perez and Jason Wang, The Effectiveness of Data Augmentation in Image Classification using Deep Learning. arXiv e-prints, art. arXiv:1712.04621 December 2017.
Eli Stevens, Luca Antiga, and Thomas Viehmann. Deep Learning with PyTorch. Manning, 2020.
10

---

Under review as a conference paper at ICLR 2021
APPENDIX
A FUNCTION CALL DEMONSTRATIONS
Automunge available for pip install:
pip install Aut omunge
Or to upgrade (we currently roll out upgrades fairly frequently):
pip install Aut omunge
upgrade
Once installed, run this in local session to initialize:
from Autnne import Aut omungee am Automunger. Autoounge ()
Then, assuming want prepare train set df train for ML, can apply default parameters
train, trainID, labels, \ validationi, validationIDI, validationlabelsll, validation2, validationID2, valicationlabelsl, \ test, testID, testlabels, labelsencoding_ featureimportance, postprocess dict = am. autune (df train)
aawellooumss_traiin finalcolumns_test,,
Note that our df train set included a labels column, we should designate the column header with the labels.column parameter. Or likewise designate any ID columns with the trainID_column parameter.
The returned postprocess dict should be saved such as with pickle.
We can then consistently prepare subsequent test data df test postmunge()):
test, testID, testlabels, Labelsencodingg_diee
postreports_dict \ am. postmunge (postprocess_ dict, df_test)
I find it helps to just copy and paste the full range of parameters for reference:
train, trainID, labels, \ validationi, vali validationIDI, validation2, validationID2, validationlabels2,, test, testID, testlabels, Labelsenncoiin_ddict,, fimalcolumsstrainn, finalcolumss_tsst
\
featureiimporancee, postprocess_dict
am. udfttrrn df_test=False, labels_collmm=Falles
trainto__columm-aalee,, testiD__colum=Faalse,, valpercent1=:0,
valperccent2=.0, htt.apappeeaaaeeeeeeeeeeeeeeeaar
hag.aaaheeea.aaaaaeee.c.acaoaaaa Lsfit==alse, numbercateegorybeerriiicc-,,,,,
MArw_marker=Faalse, Feteeeeeeeeeettreeeeeeeeeooreere featurepet=1.0, O
featuremetric=0..C, o Binaaulail,BBiaayaaaalee..e Binary=False,
PCAn case,, FCAexc1=(), Fchaocds))),eccllooofffeeeeeee excl_suffix=False,
ML_emnd = ''Linfill_type"
'defuult'' ?
"RRnndoFForree Maanfii__cmoctPPaaooooorrrttttt "randonfoorsttllassiiilrrr'

---

Under review as a conference paper at ICLR 2021
"Bandoo ;)),
'defaul'', "PCA_cmna : ()),
assigncat
:mmbr'[l, 'retn' : [], 'mnmx' :[], 'mean' [], 'MAD3': [I, '1gnm' []),
"bins' 'ssor' :[], 'pwrs': pwwrs'lll 'pwr2': :[],
'por2' [],
'bxcx' [
[addd':[]. 'sbtr': :[], 'mitp':ll 'divd':
"Log0': [I], 'logl': :[], 'Logn': 'sqrt': [[], 'rais': [],
'absl's [1,
'bnwd': :[], 'bnwk': :[],
'bnwM': 'bnwo' I[], 'bnKo's ;[],
'bnMo [1,
'bnep': [], 'bne7': []
'bne9': "bneo': 'bn7o': [],
'b99o's :1],
'bktl' :[],
'bkt2': []],
'bkt3': 'bkt4': []],
'nbr2' : :[],
'nbr3': []
'MADn': "MAD2' [[], 'tlbn' [],
'mnm2. :[],
'mmm3' [[],
"mnm4':1 'mnm5' [],
'mnm6': [],,
'ntgr' :],,
'ntg2": [],
'ntg3': 'mea2's :[],
'mea3': ;I,,
'bxc2": []],
'dxdt :[],
'd2dt': [], 'd3dt':
'dx22' ; 'd222 []],
'd3d2'3 []]
"nmdx : [],
'nmd2' []
'nmd3': 'mmdx's [[],
'mmd2': []],
'mmd3'
'shft' []],
'shf2': []
'shf3':
ssf4': ssff'' [],
'shf8''l]
'bnry' : [[],
'onht' []
'text'll 'txt2's [[],
'1010': :1] 'or10':1
'ordl" []] 'ord2 :
'ord3': 'ord4': 'Oo6'' [],
omm1'' []
'mmor':
'Unht : [[],
'Utxt' [],
'Utx2': 'Oo3'' []),
'0101':
'splt': [],
'sp12': [], 'sp15': 'sp15':
'sp19' [] 'sbst'
'sp18" []),
'sp19': [],
'sp10': 'sp16' I[), 'sp20': [], 'sbs2'
'srch' []] 'src2': [ 'src4': 'strn': [[rrn'[]] *Ingt" [], 'aggt': [ll
!mmre' [[],
'nmr2': []], 'nmem'
'nmc2' [], 'nmEU' [[,,
'nmE2':
"mr7' [[],
'nmr8': [] 'nmc7':
mmcc':ll 'nmE7' ':[], 'nmB8':l]:
oor2' :
'ors5' [],
'ors6':
ors7': 'ucet': I[] 'Uctt'
"or15':00 'or17': []], 'or19': 'or20': 'or21':l] "or22'[]]
:late':10 'dat2': [], 'dat6': 'wkdy': 'bshr's
hhddy:[]]
'wkds' []],
'wkdo's []] 'mnts': 'mnto': [[],
yyea2';[1. 'mnt2':
'mnt6':l. 'aay2' []], 'day5' [I,
"hrs2':[1. 'hrs4's : 'min2':[l, "min4':ll 'scn2':l] 'DPrt's [l,,
'DPnb':ll "DPmm ::],, 'DPbn':ll, 'DPod':l), 'DP10': 'DPoh': :
'excl':[], (exc2':1) 'exc3':l), 'exc4':l), 'exc5':l), 'exc6'; :exc6';],
"n]ll': 'copy':l), 'shfl':l), 'eval':l), 'ptfm' [1),
assignparam ['default_assiimpaarmm
(category) ('parameter)) 42)),
(category) ('cluum))) {' (parameterr : 42))),
assigninfill
('sstddiniil''' 'Miinfill'(]),
zeroinfill':ll) 'oneinfill':(],
adjinfill':l1, 'meaninfill':l]: 'medianinfill':0)
mmodeinfill'!() 'Icinfill':(1, 'naninfill':(1.
assignnan ('categories':(). 'columns': 'global': :
transformdict=()> processdict=(), evalcat-False, privacy_encode False, printstatus-True)
Or for postmunge()) with full range of parameters:
test, testID, testlabels,
Labelsencooding_dicc: postreports_dice am. ao.tommmu (postprocess dict, df if_test, Lest.D__colum False, labelscolumn = False, pandasoutput False, printstatus True, TrainlabelFFrggeeelll = False, featureeval False, driftreport False, LabelSmoothing Faise, LSfit = False, inversion = False, traindata False, returnedsets True, shuffletrain = False)
12

---

Under review as a conference paper at ICLR 2021
B ASSIGNING TRANSFORMS AND INFILL
Assigning root categories conducted assigncat parameter and assigning infill in assigninfill e. for train set df_train with column headers 'coll' and 'col2' could assign retain normalization (retn) and an integer encoding set (ntgr) with infill types zero infill and ML infill.
Note any columns don't explicitly assign will defer automation mated defaults for pass-through of other columns by passing automunge parameter powertransform 'excl'.
could turn off auto-
assigncat = { l'retn': "oatn':['coo ['coll'), 'ntgr': [ ['col2"] assigninfill ('zeroinfill': [coll'), ( ['co12']
train, trainID, labels, validationl, validationIDi, valicationlabeist, \
validation2, validationID2, valldatioonabels2,, test, testID, testlabels, Labelsencoding__iie finalcolumns_traiin
finalcolumms_test,
featureimportance, postprocess_dict \ am. adt.uttmunngee (df_train,
assigncat assigncat, assigninfill assigninfill,
pandasoutput. True)
13

---

Under review as a conference paper at ICLR 2021
C CUSTOM FAMILY TREES
Custom defined family trees of transformations can be passed to function call by way of the transformdict and processdict parameters. The transformdict parameter is used to populate family tree, and the processdict parameter is to populate supporting data structure for a new transformation categories.
Let's demonstrate scenario to assemble a transformation set in which a normalization is supple- mented by two types bin aggregation. We'll 'll create new root category 'newt' and populate with transformations pre-defined in the library. Here we'll apply an upstream retain normalization and power of ten bins, and standard deviation bins downstream of the retain normalization. We'll also include NArw transformation which designates markers for entries that were subject to infill based on values of the source column.
crans formdict
I'newt' O'parents' : ['newt '),
ssbbiing'' : [], auntssuncles' ['owr2'],
'cousin'' ['NArw'],
'children' : [],
'niecesnephews" f],
ccoorkkrr'' [],
'fiiends' ['bins']))
The corresponding processdict will make use transformation functions defined in the library for the 'retn' (retain) normalization. Here NArowtype designates the types of entries from the source column that will be targets for infill, MLinfilltype designates the types of predictive models to be trained for MI infill, and labelctgy is support entry for feature importance for cases where label in returned in multiple contigurations.
processdict { ('newt' : furctiinpointer' 'retn', 'AArwwypee' 'numeric', MMiinfilltys'' : 'numeric', 'labelctgy' 'newt'))
We can then pass these populated structures to a function call and assign a column with header 'coll' the newly defined root category 'newt'. If want also apply ML infill even this custom defined transformation set.
train, trainID, labels, \ validationi, validationIDI, valicationlabelsl, \ validation2, validationID2, validatioonabell2..
test, testID, testlabels, Labelsencoodingddict, dinalcolumss_triin finalcolumns_test,,
featureimpprtancee, postprocess_dict
am. (df_train,
assigncat ('newt': ('newt':['coll'']
assigninfill ('MLinfill'; ['coll']
transformeict transformdict, processdict processdict, pandesouttuut=True)
When comes time to process additional data, of these customizations will be saved in the returned postprocess_diee which we can then pass to the postmunge(.) function for consistent pro- cessing of a test data set.
test, testID, testlabels,
Labelsencoding_ dict, postreports_dis diet \ am. pom.postuungee (postprocess_die df_test, pandasoutput--rrue)
14

---

Under review as a conference paper at ICLR 2021
D DATA AUGMENTATION
The recommended workflow for applying training data augmentation to assign any desired target columns DP transforms in assigncat in automunge(.) call, passing any parameters to assign- param to deviate on default noise distributions if desired, and then process the same set again in postmunge() without noise injection, concatenating the results.
For the example of a received train set _triin with column headers of numeric sets 'numberl', 'number2' and column headers of categoric sets 'categoricl", "categoric2", we can elect to apply noise injections to these columns by assigning DP transforms in assigncat.
assigncat =
('DPrt': ('number1', 'number2'] 'DP10's :aall'":('caooroaaaaaa ''categoric'', 'categoric2']]
The noise injection transforms have default entries noted in READ ME for noise ratios and dis- tribution parameters. we want to overwrite for the transformation category globally can apply assignparam with default assignparam entry.
assignparam
deefault_assignparee
('DPrt' { ('sigma' 0.05, 'flip_prob' 0.5), 'DP10' p'flip_prob' 0.5)
Or we can overwrite for specific column, here we demonstrate applying scaled Laplace distributed noise instead of Gaussian to the 'DPrt' transform application to column 'number2'.
assignparam. update ('DPrt' ('number2'
Inoiseddstribbttio
"laplace')]
We can then process our train set in automunge() with noise injection. Here passing the parameter labels_column True signals that the final column is be returned in the labels set.
train, trainID, labels, \ validationl, validationIDI, validationlabelsl, validation2, validationID2, validationlabelss
test, testID, testlabels, lebel final col columsstaai,,, finalcolumns_ter
featureimportance, postprocess_dice \ am, aut omunge (df_train,
labels_column True, assigncat assigncat, assigmparam assignparam, pandasoutput True, printstatus False)
The convention for noise injection transforms in the DP family is that noise is injected train sets by default and not to test sets unless signaled by the traindata parameter in postmunge()). Test sets passed to automunge(.) do not receive injections.
Once the training data set has been processed with noise, the same data set can be processed again the postmunge()) function and concatenated, using the postprocess dictionary returned from the corresponding automunge(.) call. The postmunge accepts parameter traindata which signals to the DP whether to inject noise in postmunge(.). traindata is Boolean defaulting to False for no noise injection. When True postmunge(.) will inject noise. Note that each application will inject a unique randomization of noise that is independent of the randomseed parameter although consistent with any distribution parameters.
15

---

Under review as a conference paper at ICLR 2021
rest, testID, testlabels, Labelsencoding_diee postreports, dict - am. wun.postmungg (postprocess_daee _icit, df_train, pandasoutput True, traindata False, printstatus False)
We can then concatenate the results for the train data, ID sets, and label sets to double the number of samples include with and without noise.
train = pd. concat [[train, test], axis=0,
trainID = concat ([rainnI,, testID), axis=0, ignore_iniene index=True) labels pd. concat [[aablls, testlabels), axis=0, ignore index=True)
Although in regular use the automunge(.) function is intended as a resource to apply a train test split based the valpercent1 and valpercent2 parameters, with data augmentation recommend extracting validation entries prior to passing a df train so that the same train can be redundantly encoded with without noise in adjacent calls. The extracted validation set can be processed separately in postmunge()) or can be passed as a destest to automunge(.).
Note that while our experiments only doubled the number of samples to include with and with- out noise injection, further multiples of noise injected samples may also be applied by processing in postmunge(() with traindata=True. The noise distribution properties will be consistent with any parameters passed in the corresponding automunge(.) call. In our experiments we found that too much redundancy had potential to interfere with training convergence.
In an alternate workflow, when only doubling of the number of samples is desired, the processing can all be performed in a single automunge()) call by passing the same df. train set as both the train and test sets as follows:
train, trainID, labels, \ validationi, validationIDl, valicationlabelsl, validation2, validationID2, validationlabels2, test, testID, testlabels, labelsencoding_dicct finalcolumn__tree
finalcolumns_test:
featureimportance, postprocess_dict \ am, aut.auutmungeeeeee (df_train,
df_test = df_train,
labels_colume column True,
assigncat = assigncats
assignparam assigmparam, pandasoutput True, printstatus False)
train pd. concat ([train, test], axis=0, ignore_ index=True)
trainID = pd. concat ([trainID, testID], axis=0, ignore_ (nnore_index=Tree]
labels = pd.conttt [[labels, testlabels), axis=0, ignore_index=True)
16

---

Under review as a conference paper at ICLR 2021
E COLUMN TYPES OF RETURNED DATA
The data returned from an automunge()) call is intended to be suitable for direct application of machine learning in the framework of user's choice. In some cases, downstream machine learning libraries may accept as input designations for the types of data found in a column, such as for instance if column contains numeric or categoric set. These type of designations may be used for determination of whether to apply an entity embedding layer for instance.
The automunge(.) function thus returns report of data types for returned columns, available in the returned dictionary as postprocess dict[ 'columntype report").
The report lists column headers of the retuned columns, aggregated by different types of column contents. Specifically, it lists column headers for column content types:
continuous
boolean
ordinal
onehot
onehot_sets
binary
binary sets
passthrough
Here the onehot aggregation contains all one-hot encoded columns, while the onehot sets sets sub- aggregates those same columns for those that originated from the same transformation, and similarly with the binary and binary-sets aggregations.
17

---

Under review as a conference paper at ICLR 2021
F INVERSION
The Automunge library includes an inversion option to recover the form of data as prior to trans- formations. This type of operation might be useful for instance in converting ML predictions back to the original form of labels, or otherwise for recovering data to invert transformations. The in- version operation is performed The method relies on data properties stored in the postprocess_dice which was returned from the corresponding automunge(.) call. For cases where source column is included in multiple configu- rations, the inversion operation relies heuristic of selecting the shortest path of transformations with full information retention.
the postmunge(.) function by activating the inversion parameter.
As an example of an inversion operation, if we want to recover the form of labels after a ML predic-- tion process, we can pass those predictions as dataframe or array as:
invert, recovered_ am. postmunge (postprocess_dicct predictions, inversion= 'labels', Label mabblllsmothhinefaalse pendasoutput=True, printstatus=True)
inversion info info__ictt dict
Similarly, if we want to recover the form of training or test dataset for transformation inversions, we can pass as:
åf_invert, recovered_list, itversiio__ifooddit \
am postmunge (postprocess_diiee _dctt, train, inversion= "test',, pandasoutput printstatus-True)
18

---

Under review as a conference paper at ICLR 2021
G EXPERIMENT DETAILS
Further details on the experiments are provided to demonstrate statistical significance. The Tables and reported averages of performance metrics with comparison the raw data scenario. Here ad- ditional detail is provided including standard deviation of the metrics and sampled repetition count serving as the basis. We suspect a large contributor the variance originates from the stochastic sampling of data set partitions in the experiments, as would explain the increasing standard devia- tions with smaller training set size.
Table 11: Higgs Experiment Details (AUC Average Standard Deviation // Repetitions)
Retain w/ Noise full
Retain Noise partial
Retain with Bins
Retain w/ Augment
Raw Data
7-Score
Retain
full data St Dev
0.8670 0.0002 6
0.8668
0.8667 0.0005 6
0.8660 0.0013 6
0.8268 0.0007 5
0.8624 0.0007 5
0.8667
0.0004 6
0.0015
Repetitions
6
5% data Dev
0.8436
0.8432
0.8431
0.8433
0.7679
0.8388
0.8453
0.0011
0.0011
0.0016
0.0014
0.0016 30
0.0011 30
0.0006
Repetitions
30
30
30
30
30
0.25% data
0.7715
0.7716
0.7716
0.7718
0.7052
0.7668
0.7821
0.0024
0.0025
0.0026
0,0029
0,0034
0.0025
0,0020
Repetitions
100
100
100
100
100
100
100
Table 12: Higgs SVC Experiment Details (Accuracy Average // Standard Deviation // Repetitions)
Retain w/
Retain w/ Noise partial
Retain w/ Augment partial
Retain with Bins
Raw Data
Z-Score
Retain
Noise full
Accuracy St Dev
0.6410 0.0003 6
0.6410 0.0003 6
0.6410 0.0003
0.6821 0.0005
0.6201 0.0006 6
0.6393 0.0003 6
0.6402
0.0003
Repetitions
19

---

Under review as a conference paper at ICLR 2021
H A FEW HELPFUL HINTS
A few highlights that might make things easier for first-timers:
1) automunge()) returns sets numpy arrays by default (for universal compatibility with ML plat- forms). A user can instead receive the returned sets as pandas dataframes by passing the parameter pandasoutput True
2) Even the sets are returned as numpy arrays, you can still inspect the returned column headers with the returned list we demonstrate as finalcolumns ttrin
3) Printouts are turned on by default, they can be turned off with printstatus=False
4) Note for data sets with just a few rows, such as those demonstrated here, there is PCA heuristic to apply dimensionality reduction when the number of features is more than 50% of the number of observations in the train set (this is a somewhat arbitrary heuristic). This can be turned off with ML cmnd = { PPCA type":'off")
5) Speaking of PCA, you do want to apply PCA, useful option allows you exclude from dimensionality reduction boolean or ordinal encoded columns, available with ML.cmnd ('PCA.cmnd': :bbool_orddd :CAexel':Truee
6) Note that data shuffling is on by default for the train set and off by default for the test sets returned from automunge()) and postmunge().. you want to shuffle the test data in automunge too you can pass shuffletrain 'traintest'. Or to shuffle the test data returned from postmunge you can pass the postmunge parameter shuffletrain True.
7) The automated feature importance evaluation is easy to use, you just need to be sure to designate a label column with labels column 'column header'. Then just pass featureselection = True and printouts will return results well as the returned report featureimportance
8) To ensure that you can later prepare additional data for inference, please be sure to save the returned postprocess_dict such as with pickle library.
9) Importantly, when you pass a train set to automunge(.) please designate any included labels column with labels coluumncommmmmmme = (label column header string), which may be an integer index for numpy arrays. When you go to process additional data in postmunge, the columns must have consistent headers those originally passed to automunge. Or if you originally passed numpy arrays, just be sure that the columns are in the right order. If you're 're passing postmunge()) data that includes a column originally designated as a label to automunge, just apply labelscolumn True.
10) Speaking of numpy arrays, you pass numpy arrays instead of pandas dataframes to the func- tion, all of the column assignments will accept integers for the column number.
11) When applying ML infill, which is based on Scikit-Learn Random Forest implementations, useful ML_emnd if you don't mind little more training time is to increase the number of timators as e.g. ML_cmnn
-nniiiiillmddd'::.... "RandomForrstClaaiffee" of " FaaddomForesRRgrresoor :f 1ottimmatoss:100000
12) Note that any columns you want to exclude from processing, you can either assign them to root category 'excl' in assigncat if you don't mind their retention (noting that they will still be included in ML infill so will need numerical encoding), or you can carve them out from the set to be returned in ID sets consistently shuffled and partitioned such as between training and validation data, just pass a list of column headers to trainID column and/or testID column. You can also turn off the automated transforms and only perform those designated assigneat by passing powertransform=
I INTELLECTUAL PROPERTY DISCLAIMER
Automunge is released under GNU General Public License v3.0. Full license details available on (anonymized). Contact available via (anonymized). Copyright (C) 2020 All Rights Reserved. Patent Pending, applications (anonymized)
20


Reviews:
[{"review_id": "eN-1gZc4tsA", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a number of data preprocessing options for numeric features provided by an open source library automunge. The most of the paper focuses on explaining the specific transformations offered under each option, including normalization, binning, and noise injection. For normalization, a new transformation 'retain' is offered in addition to traditional z-score, min-max etc. The paper uses one section between normalization and binning to explain a notion 'family tree primitives' which is used in the composition of multiple transformations. In experiments, the paper uses the higgs dataset. Three settings of the dataset are used: full data, 5% data and 0.25% data. 6 settings of transformations are used: raw data, z-score, retain, retain with bins, retain with noise injection, and retain with partial noise injection. When averaged over the three settings of the dataset, using raw data leads to suboptimal auc score compared with the other five. It is acknowledged that \"the metrics for normalized data were slightly better than raw on average, it is not clear if they were sufficiently statistically significant to draw firm conclusions. \" The paper concludes that \"any consideration around benefits of feature engineering should distinguish\nfirst by scale of data available for training. When approaching big data scale input with infinite\ncomputational resources there may be less of a case to be made for much beyond basic normalized\ninput. \"\n\nPros: The code is open sourced and a larger number of data preprocessing options are offered. The library is released as a python package which is easy to access. In the experiments there are certain cases where the preprocessing helps.\n\nCons: \n\n1. The experiment results do not convincingly demonstrate the utility of the proposed library. First, when using full data of higgs, the auc is highest when using the raw data. It is even better than using z-score or retain to normalize the data. That is contradictory with the suggestion to consider even basic normalization. Second, when small samples of the data are used, the best auc score among all studied transformations is much lower than the auc score on full raw data. That means even with the transformations, the accuracy does not reach on par with simply using full raw data. Then it raises the question whether the setting of using small sample of higgs data matters. Third, even if we do care about those settings, the best performing transformation is z-score only. That is not a contribution of this paper. To conclude, it is not surprising that one can find some scenarios where some transformation improves the accuracy, but the paper has not presented a convincing scenario where the proposed library is useful.\n\n2. The explanation of 'family tree primitives' confuses me. Table 3 does not help as it is full of unexplained terms. Without examples and context it is difficult to know what 'parents' 'siblings' etc. refer to.\n\n3. It remains a question how a user should select transformations from the numerous offered options.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Qj-L5s0CTJo", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper reviews an existing Python feature engineering library, Automunge. It describes Automunge's functions operating on numeric (as opposed to categorical) input data. It describes notions of transformations, operations available to bin, inject noise, process sequential data and integer sets. \n\nThe Automunge library implements what the data scientist might expect from a library of feature transforms for numerical data. It is not particularly rich or original. The library structure is not made clear in the paper. It is a pity that the paper does not compare to any other feature engineering library, although there are very many available. \n\nThe experiments section reports two experiments on one dataset; it reports AUC and accuracy metrics under different feature pre-processing: this reporting is hardly relevant and does not demonstrate any interesting property of Automunge. For instance, it would be interesting to see whether Automunge is particularly flexible, clear, or convenient when one wants to implement chains of transformations. It is not clear why, table 8, it is interesting to observe 100%, 5% and 0.25% of data regimes, nor why we should average over these three.\n\nTables 3 and 5, and none of the figures are mentioned in the text, let alone commented upon. The figures are left unexplained and contain undefined abreviations (NArw, DPo3, DPo6). I could not make sense of them. The family tree structure which seems to be the object of Table 3 is not explained, and I could not make sense of it, though it seems to be important. \n\nIn table 4, I could not understand what the columns refer to; equally, the transformations mentioned there are left undefined: for instance, in case, as I assume, \"Number of standard deviations from the mean\" is $x \\mapsto round(|(x_i - µ) / \\sigma) |)$ (or maybe the absolute value needs to be replaced by round brackets? this is left ambiguous), it should be properly defined. Similarly, what is \"Powers of ten\"? Does it mean $x \\mapsto round(\\log_{10} x) $ ? The id strings are cryptic. It is not clear why identifiers seem to be all limited to four characters; this makes the entire naming scheme very difficult to follow, as abreviations seem arbitrary; in addition, this is contrary to Python variable naming conventions, particularly naming conventions in popular and successful ML libraries, which prefer explicit and long-form function and argument naming over abreviated and cryptic ones.\nTable 6 refers to categoric noise injections, but the paper was meant to be about numeric input variables, so I'm not clear what it is doing here.\n\nThe text is hard to understand. This is in part, but not only, due to long-winded, unclear sentences: for instance the very first sentence is obscure and ill-structured, with several syntax errors, for instance the repeated and often incorrect usage of \"such as\". Throughout the text, syntax issues make the text hard to understand. Examples of this:\n- \"data transformations ... are to be directed for application to a distinct feature set as input\"\n- such as multiplicative properties at, above, or below.\n- potentially including custom defined transformation functions with minimal requirements of simple data structures\n- the last sentence of sec3\n- the first sentence of sec4 is needlessly intricate and seems to say nothing more than \"Automunge transformations are invertible\".\n- the paragraph below figure 2\n- a given ratio of input entries are flipped to one of the other encodings between which have a uniform probability\n\nThe very last conclusion paragraph is surprising and seems irrelevant. Similarly, the paper mentions quantum computation for reasons that escape my comprehension, section 2.\n\nThe paper is so hard to understand that it does not allow one to evaluate possible upsides of Automunge on its own merit. After quite some hesitation, I have to assign this paper a severe rating due to the conjunction of several severe shortcomings.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KYHZ5KebbFa", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nSummary:\n\nThe paper describes a library (Automunge) for pre-processing tabular data to prepare the data for downstream machine learning tasks. The paper also describes how to use the said library and the various options (including some new forms of normalization) available in the library. Experimental evaluation on Higgs Boson interaction dataset is provided, following the experimentation in Baldi, Sadowski and Whiteson 2014. Some improvements over the published results in Baldi 2014 are shown. \n\nKey strengths:\n\nThe paper is well written for the most part, apart from a few typos. The appendix provides examples of usage of the proposed library. The supplementary material contained the code and an extensive readme for the code. \n\nSuggestions for improvements:\n\n1. The primary novelty of the library is to ease the burden of tabular data pre-processing. However, it seems that the user has to specify the configuration for the pre-processing unless the default options are acceptable. For most machine learning tasks, the choice of pre-processing can be heavily influenced by the context and the domain expertise. It is unclear how a library can automatically derive these options correctly, without the knowledge of the downstream machine learning task (or indeed the domain). \n2. The paper provides evaluation on one dataset (Higgs Boson interactions) to support the claim that the library is of value for many machine learning tasks. The setup of the experiment is slightly different from the original paper (Baldi 2014): i.e. learning rate schedule is different, but network setup is the same. Looking at the results in the original paper (Table 3 of Supp material in Baldi 2014), the results presented in this paper are in the same range as the original paper (but also within the variance of results in the original paper), so it's difficult to extrapolate how significant the improvement due to retain normalization (this paper) is, and how it might generalize to other machine learning tasks, or even other datasets. Also, some results improve upon the original paper, whereas others don't, which makes it harder to say how statistically significant the effect of the pre-processing presented in this paper is. \n3. Retain normalization: The paper claims that retain normalization is better because it retains the sign of the original datapoints, but it's unclear why this is important if it doesn't affect the eventual result. Perhaps this could be clarified? \n4. Software review: The entire library is written in a single Python file. The naming convention is not Pythonic. From the usage examples and reviewing the code, it's clear that the library makes little use of encapsulation of data structures, leading to unwieldy usage. All documentation is provided as a single Readme, rather than as Sphinx documentation or API documentation. There don't appear to be any tests or continuous integration included. The code is not modularised. There are no docstrings with param/return descriptions or type annotations. \n\nOverall comments:\n\nFor what is mainly a description of an automated data pre-processing library, the usage of the library is not sufficiently automatic (i.e. setting up the options amounts to feature engineering, which the paper claims to circumvent). The new normalization technique presented in the paper does not yet demonstrate a clear improvement over current options. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "DsaBzmjFmPu", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a set of numeric normalization features from the Automunge open-source python library, and evaluates them in two machine learners (one neural, one SVM) on a data set.  In general the paper discusses standard normalizers rather than introducing novel techniques, and the experiments are too limited to evaluate the effectiveness of the approaches.\n\nI did not understand the family tree primitives, unfortunately.  I think I need a concrete example.  What do upstream and downstream mean in this context?  What is a “generation”?  Why are the primitive names (the familial relations) appropriate to describe each row in Table 3?\n\nThe experiments also do not show that the transformations described in the paper offer benefits for neural models.  In fact, on the one data set and one neural architecture evaluated, using the best normalizations results in essentially the same performance as using raw unnormalized data.  Even if the experimental results had been positive, the limited scope of the investigation would make it impossible to draw general conclusions.  A substantially expanded set of experiments that looked at more neural architectures and many more data sets might give practitioners more useful guidance, especially if it led to concrete recommendations about which normalizations to try for which type of data.\n\nMinor:\n\nclassical “or quantum” computation may require non-negative feature values?  Why mention quantum here?\n\nTables 8 -> Table 8\n\nThe author refers to Automunge as “our” library at one point, breaking anonymity.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: While I'm sure there are many merits to the underlying work here, the consensus of the reviews is to recommend a rejection as an ICLR paper. That recommendation is based on issues with significance as well as on clarity issues, noted by reviewers even after the revisions.

One pattern I noticed was that it seemed unclear whether the paper was to be regarded primarily as a software paper or as a paper on preprocessing. Most initial reviews evaluated it primarily as a software paper, but some comments from the authors in the discussion period seemed to frame it instead as a paper about research on preprocessing (independent of software). See my other comment for more detail on this question.

Regardless of the intended framing, on significance as an ICLR submission, reviewers did not support its acceptance by either standard:
* R1 post-response: "Regarding how to frame the paper (either about feature pre-processing or the software library), my (favourable) interpretation is to frame it as about the software library. As a paper about feature pre-processing it would have even less merit."
* R3 post-response: "the work has more upside in the software contribution than the feature pre-processing research"
* R4 post-respones: "After reading the revised version and the author response, I am still not convinced that the paper makes a substantial contribution either on the fundamental research angle or the software library angle"

One specific issue was that the experiments did not adequately support the main claims:
* R2 post-response: "I feel that the experiments in Section 8 are still too limited to demonstrate the value of the techniques and software"
* R4 post-response: "While the main contributions of the paper are still a little ambiguous, the experiments don't seem to support the claims (ease of use of the library). The results seem to suggest that one of the contributions is the improved accuracy of results, but then the experimentation is far too limited to draw such a conclusion."

On clarity:
* R1 post-response: "The new section 5 is still very hard to understand and I still couldn't make sense of the family tree primitive mechanism"
* R3 post-response: "I agree with AnonReviewer1 about the difficulty of understanding the family tree primitives, both before and after the revision"
* R2 post-response: "I am still unclear on the family tree primitives [...] it's not clear to me what problem the family tree primitives solve"


The authors showed a lot of enthusiasm and good spirits in working to improve the submission. I hope the feedback provided here is useful.

However, based on the consensus of the reviews, I recommend rejection of this submission.
Decision: Reject