Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 2: Strong rejection
review_content: This paper reviews an existing Python feature engineering library, Automunge. It describes Automunge's functions operating on numeric (as opposed to categorical) input data. It describes notions of transformations, operations available to bin, inject noise, process sequential data and integer sets. 

The Automunge library implements what the data scientist might expect from a library of feature transforms for numerical data. It is not particularly rich or original. The library structure is not made clear in the paper. It is a pity that the paper does not compare to any other feature engineering library, although there are very many available. 

The experiments section reports two experiments on one dataset; it reports AUC and accuracy metrics under different feature pre-processing: this reporting is hardly relevant and does not demonstrate any interesting property of Automunge. For instance, it would be interesting to see whether Automunge is particularly flexible, clear, or convenient when one wants to implement chains of transformations. It is not clear why, table 8, it is interesting to observe 100%, 5% and 0.25% of data regimes, nor why we should average over these three.

Tables 3 and 5, and none of the figures are mentioned in the text, let alone commented upon. The figures are left unexplained and contain undefined abreviations (NArw, DPo3, DPo6). I could not make sense of them. The family tree structure which seems to be the object of Table 3 is not explained, and I could not make sense of it, though it seems to be important. 

In table 4, I could not understand what the columns refer to; equally, the transformations mentioned there are left undefined: for instance, in case, as I assume, "Number of standard deviations from the mean" is $x \mapsto round(|(x_i - µ) / \sigma) |)$ (or maybe the absolute value needs to be replaced by round brackets? this is left ambiguous), it should be properly defined. Similarly, what is "Powers of ten"? Does it mean $x \mapsto round(\log_{10} x) $ ? The id strings are cryptic. It is not clear why identifiers seem to be all limited to four characters; this makes the entire naming scheme very difficult to follow, as abreviations seem arbitrary; in addition, this is contrary to Python variable naming conventions, particularly naming conventions in popular and successful ML libraries, which prefer explicit and long-form function and argument naming over abreviated and cryptic ones.
Table 6 refers to categoric noise injections, but the paper was meant to be about numeric input variables, so I'm not clear what it is doing here.

The text is hard to understand. This is in part, but not only, due to long-winded, unclear sentences: for instance the very first sentence is obscure and ill-structured, with several syntax errors, for instance the repeated and often incorrect usage of "such as". Throughout the text, syntax issues make the text hard to understand. Examples of this:
- "data transformations ... are to be directed for application to a distinct feature set as input"
- such as multiplicative properties at, above, or below.
- potentially including custom defined transformation functions with minimal requirements of simple data structures
- the last sentence of sec3
- the first sentence of sec4 is needlessly intricate and seems to say nothing more than "Automunge transformations are invertible".
- the paragraph below figure 2
- a given ratio of input entries are flipped to one of the other encodings between which have a uniform probability

The very last conclusion paragraph is surprising and seems irrelevant. Similarly, the paper mentions quantum computation for reasons that escape my comprehension, section 2.

The paper is so hard to understand that it does not allow one to evaluate possible upsides of Automunge on its own merit. After quite some hesitation, I have to assign this paper a severe rating due to the conjunction of several severe shortcomings.
review_title: This work may be interesting, but the paper doesn't show why

final_score: rating: 3: Clear rejection
review_content: 
Summary:

The paper describes a library (Automunge) for pre-processing tabular data to prepare the data for downstream machine learning tasks. The paper also describes how to use the said library and the various options (including some new forms of normalization) available in the library. Experimental evaluation on Higgs Boson interaction dataset is provided, following the experimentation in Baldi, Sadowski and Whiteson 2014. Some improvements over the published results in Baldi 2014 are shown. 

Key strengths:

The paper is well written for the most part, apart from a few typos. The appendix provides examples of usage of the proposed library. The supplementary material contained the code and an extensive readme for the code. 

Suggestions for improvements:

1. The primary novelty of the library is to ease the burden of tabular data pre-processing. However, it seems that the user has to specify the configuration for the pre-processing unless the default options are acceptable. For most machine learning tasks, the choice of pre-processing can be heavily influenced by the context and the domain expertise. It is unclear how a library can automatically derive these options correctly, without the knowledge of the downstream machine learning task (or indeed the domain). 
2. The paper provides evaluation on one dataset (Higgs Boson interactions) to support the claim that the library is of value for many machine learning tasks. The setup of the experiment is slightly different from the original paper (Baldi 2014): i.e. learning rate schedule is different, but network setup is the same. Looking at the results in the original paper (Table 3 of Supp material in Baldi 2014), the results presented in this paper are in the same range as the original paper (but also within the variance of results in the original paper), so it's difficult to extrapolate how significant the improvement due to retain normalization (this paper) is, and how it might generalize to other machine learning tasks, or even other datasets. Also, some results improve upon the original paper, whereas others don't, which makes it harder to say how statistically significant the effect of the pre-processing presented in this paper is. 
3. Retain normalization: The paper claims that retain normalization is better because it retains the sign of the original datapoints, but it's unclear why this is important if it doesn't affect the eventual result. Perhaps this could be clarified? 
4. Software review: The entire library is written in a single Python file. The naming convention is not Pythonic. From the usage examples and reviewing the code, it's clear that the library makes little use of encapsulation of data structures, leading to unwieldy usage. All documentation is provided as a single Readme, rather than as Sphinx documentation or API documentation. There don't appear to be any tests or continuous integration included. The code is not modularised. There are no docstrings with param/return descriptions or type annotations. 

Overall comments:

For what is mainly a description of an automated data pre-processing library, the usage of the library is not sufficiently automatic (i.e. setting up the options amounts to feature engineering, which the paper claims to circumvent). The new normalization technique presented in the paper does not yet demonstrate a clear improvement over current options. 
review_title: A new library for tabular data pre-processing for machine learning tasks 

final_score: rating: 3: Clear rejection
review_content: This paper presents a set of numeric normalization features from the Automunge open-source python library, and evaluates them in two machine learners (one neural, one SVM) on a data set.  In general the paper discusses standard normalizers rather than introducing novel techniques, and the experiments are too limited to evaluate the effectiveness of the approaches.

I did not understand the family tree primitives, unfortunately.  I think I need a concrete example.  What do upstream and downstream mean in this context?  What is a “generation”?  Why are the primitive names (the familial relations) appropriate to describe each row in Table 3?

The experiments also do not show that the transformations described in the paper offer benefits for neural models.  In fact, on the one data set and one neural architecture evaluated, using the best normalizations results in essentially the same performance as using raw unnormalized data.  Even if the experimental results had been positive, the limited scope of the investigation would make it impossible to draw general conclusions.  A substantially expanded set of experiments that looked at more neural architectures and many more data sets might give practitioners more useful guidance, especially if it led to concrete recommendations about which normalizations to try for which type of data.

Minor:

classical “or quantum” computation may require non-negative feature values?  Why mention quantum here?

Tables 8 -> Table 8

The author refers to Automunge as “our” library at one point, breaking anonymity.

review_title: Concerns with clarity, novelty, scope of experiments

final_score: rating: 2: Strong rejection
review_content: This paper introduces a number of data preprocessing options for numeric features provided by an open source library automunge. The most of the paper focuses on explaining the specific transformations offered under each option, including normalization, binning, and noise injection. For normalization, a new transformation 'retain' is offered in addition to traditional z-score, min-max etc. The paper uses one section between normalization and binning to explain a notion 'family tree primitives' which is used in the composition of multiple transformations. In experiments, the paper uses the higgs dataset. Three settings of the dataset are used: full data, 5% data and 0.25% data. 6 settings of transformations are used: raw data, z-score, retain, retain with bins, retain with noise injection, and retain with partial noise injection. When averaged over the three settings of the dataset, using raw data leads to suboptimal auc score compared with the other five. It is acknowledged that "the metrics for normalized data were slightly better than raw on average, it is not clear if they were sufficiently statistically significant to draw firm conclusions. " The paper concludes that "any consideration around benefits of feature engineering should distinguish
first by scale of data available for training. When approaching big data scale input with infinite
computational resources there may be less of a case to be made for much beyond basic normalized
input. "

Pros: The code is open sourced and a larger number of data preprocessing options are offered. The library is released as a python package which is easy to access. In the experiments there are certain cases where the preprocessing helps.

Cons: 

1. The experiment results do not convincingly demonstrate the utility of the proposed library. First, when using full data of higgs, the auc is highest when using the raw data. It is even better than using z-score or retain to normalize the data. That is contradictory with the suggestion to consider even basic normalization. Second, when small samples of the data are used, the best auc score among all studied transformations is much lower than the auc score on full raw data. That means even with the transformations, the accuracy does not reach on par with simply using full raw data. Then it raises the question whether the setting of using small sample of higgs data matters. Third, even if we do care about those settings, the best performing transformation is z-score only. That is not a contribution of this paper. To conclude, it is not surprising that one can find some scenarios where some transformation improves the accuracy, but the paper has not presented a convincing scenario where the proposed library is useful.

2. The explanation of 'family tree primitives' confuses me. Table 3 does not help as it is full of unexplained terms. Without examples and context it is difficult to know what 'parents' 'siblings' etc. refer to.

3. It remains a question how a user should select transformations from the numerous offered options.
review_title: Experiments do not convince

Meta-review: While I'm sure there are many merits to the underlying work here, the consensus of the reviews is to recommend a rejection as an ICLR paper. That recommendation is based on issues with significance as well as on clarity issues, noted by reviewers even after the revisions.

One pattern I noticed was that it seemed unclear whether the paper was to be regarded primarily as a software paper or as a paper on preprocessing. Most initial reviews evaluated it primarily as a software paper, but some comments from the authors in the discussion period seemed to frame it instead as a paper about research on preprocessing (independent of software). See my other comment for more detail on this question.

Regardless of the intended framing, on significance as an ICLR submission, reviewers did not support its acceptance by either standard:
* R1 post-response: "Regarding how to frame the paper (either about feature pre-processing or the software library), my (favourable) interpretation is to frame it as about the software library. As a paper about feature pre-processing it would have even less merit."
* R3 post-response: "the work has more upside in the software contribution than the feature pre-processing research"
* R4 post-respones: "After reading the revised version and the author response, I am still not convinced that the paper makes a substantial contribution either on the fundamental research angle or the software library angle"

One specific issue was that the experiments did not adequately support the main claims:
* R2 post-response: "I feel that the experiments in Section 8 are still too limited to demonstrate the value of the techniques and software"
* R4 post-response: "While the main contributions of the paper are still a little ambiguous, the experiments don't seem to support the claims (ease of use of the library). The results seem to suggest that one of the contributions is the improved accuracy of results, but then the experimentation is far too limited to draw such a conclusion."

On clarity:
* R1 post-response: "The new section 5 is still very hard to understand and I still couldn't make sense of the family tree primitive mechanism"
* R3 post-response: "I agree with AnonReviewer1 about the difficulty of understanding the family tree primitives, both before and after the revision"
* R2 post-response: "I am still unclear on the family tree primitives [...] it's not clear to me what problem the family tree primitives solve"


The authors showed a lot of enthusiasm and good spirits in working to improve the submission. I hope the feedback provided here is useful.

However, based on the consensus of the reviews, I recommend rejection of this submission.
Decision: Reject