Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: This paper studies the interpretability of neural networks. Specifically, the authors use four approaches with different objectives and regularizations and use cMLSGA, a genetic optimization method to optimize these problems. Eventually, the authors conduct experiments to validate that Fit to Median Error measure possesses better interpretability compared to L1, L2 and dropout regularizations since it regularizes the learnt input-output relationships to the conditional median. 
main_review: This is a pretty empirical work. The structure of this paper is not conventional but still retains its own logic.
limitations_and_societal_impact: The authors want to validate that the Fit to Median Error is more interpretable by comparing its performance to other different measures since the authors think that more consistent or accurate models have more interpretability. This logic may not be valid since an interpretability model should tell us why this model makes this prediction. Lipton [1] discussed that interpretability can be achieved if we human can simulate the decision-making process of the model in a limited time. Kim et al. [2] stated that interpretability is the degree to which a human can consistently predict the model’s result. There are also other definitions of interpretability such as interpretability is the degree to which a human can understand the cause of a decision [3]. Theses three are all consistent with what I mentioned, i.e., “an interpretable model should tell us why this model makes this prediction”. For example, a decision tree will tell you why it makes that decision since the final decision is based on the splitting of features on the node of the tree. Merely making the prediction more accurate is not enough for interpretability.

In addition, I think only one dataset may not be sufficient to convince the readers. The authors should also provide more details of the model and method by such as providing formulas and structure of the model.

[1] Z. C. Lipton, “The mythos of model interpretability,” CoRR, vol. abs/1606.03490, 2016.
[2] B. Kim, O. Koyejo, and R. Khanna, “Examples are not enough, learn to criticize! criticism for interpretability,” in NIPS, 2016, pp. 2280–2288.
[3] T. Miller, “Explanation in artificial intelligence: Insights from the social sciences,” Artificial Intelligence, vol. 267, pp. 1–38, 2019.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper claims that building ML models through automation can become a black-box approach that results in lower errors but also lower interpretability. Also, minimizing traditional error measures cannot guarantee an accurate approximation of the ground truth. The paper studies using fit to Median Error measure to produce better model with regard to interpretability of models. 

The paper uses ship power prediction problem that aims to reduce fuel consumption, a challenging regression problem, as an application where fit to Median Error measure is used. The paper compares four different genetic algorithms settings: minimizing the Maximum Absolute Error of the networks, minimizing the Maximum Absolute Error and minimizing the Mean Fit to Median. 

The experiment results demonstrate that all four settings produced similar Mean Absolute Errors from networks on their Pareto fronts. But the Mean Fit to the Median setting obtained much better interpretability (fit to the ground truth). 

main_review: 
The paper is mostly about applying existing techniques to an application. The GA, fully-connected neural network, and different measures are not new ideas. The paper cites many other works.

The paper makes quite many claims. Some of them are quite difficult to comprehend, For instance, in page 8, the paragraph states "Across all four approaches, the genetic algorithm producing networks with the highest Mean Absolute Error is the approach which does not provide extra weighting to sparse areas of data." What is "extra weighting"? The conclusion that GAiii is the approach with both accurate and consistent learnt speed-power curves is a bit stretchy. 


It is interesting to read the paper. However, it can be better written. For example, the section title of "3 cMLSGA Parameters" can be "Genetic Algorithm Software cMLSGA Parameter Setting"

The paper uses an application to prove that fit to Median Error measure produces more interpretable model. I found it is not too convincing. The title of paper is about automating interpretable ML through a comparison of loss functions to regularizers. Based on the methodology and experiment results, it is not clear to me that the goal is achieved. 


limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 3: Clear rejection
review_content: summary: In this paper the authors utilize a new Fit to Median Error measure for a practical machine learning problem of predicting the shaft powering of a vessel using genetic algorithms. They complete an empirical study using different combinations of loss functions and genetic algorithms to demonstrate that the choice of Fit to Median error improves both model accuracy and interpretability. 

main_review: The paper presents a complete empirical study. That said, the reviewer considers the paper lack of novelty and is discussing a very specific applied problem that might not be of the general interests of NeurIPS audience. 

The major reasons that lead to the reviewer's rating are as follows.

1. The paper does not define a clear research question. 

2. The paper lacks proper related research. Two key concepts covered in the paper are Fit to Median Error and Genetic Algorithms (GAi-Gv) whereas neither of them is novel or clearly elaborated. Therefore it is hard to justify the use of this new error measure, hence the novelty of the paper. 

3. The paper does not have enough details to demonstrate how the conclusions or techniques in the paper can be generalized to other machine learning questions. '

====
Post rebuttal:

The reviewer would like to thank the authors for their response, and would like to keep the initial rating.

limitations_and_societal_impact: N/A
review_title: null

Meta-review: This paper makes a mistake about interpretable machine learning, which is quite often on this topic, in that the interpretation involves many aspects other than accuracy. It requires a major revision of the paper to better reflect the content, thus cannot be accepted in the current form.
Decision: Reject