Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: This paper proposes incorporating (a large amount of) human knowledge into policy network design such that it solves the problem directly or within a few iterations of training. The network is trained by the proposed Self Reward Design (SRD) mechanism to simply consolidate its greedy decision, which as claimed by the authors would help supplement the human design of the network.
main_review: Strengths:
- Throughout the paper, the only thing that makes sense in my opinion is that the goal of incorporating human knowledge into the network design. 

Weaknesses:
- Terrible writing. Really hard for readers to understand. Lots of unexplained notations and technical details but much less illustration of the intuition.
- Unreasonable technical approach. Why does the network need to be trained if the goal is just mimicking the greedy action? Can we just use the greedy output instead? 
- Hard to generalize. The network is designed case by case with heavy human effort, I don't see any learnable component in the network that makes sense and effectively supplements the human design.
- Unconvincing results. This is probably because the technical approach is not reasonable. SRD fails to improve the performance in general by looking at the result figures.
summary_of_the_review: A clear rejection based on the weaknesses mentioned above and no intuition to the community.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes using individually designed, interpretable neural networks to solve a given task. Each neuron in the network is hand-designed to serve a specific task-dependent purpose. These hand-designed solutions are then optimized using environment interactions. The neuron level design of the network is used to provide human interpretability, as the behavior of the network can be understood based on the activations during environment interactions. The neural network structure of the policy can then be optimized via gradient descent to provide a performance improvement over the hand-designed solution.
main_review: Interpretability for deep reinforcement learning is an important and significant topic of research. This paper suggests a mechanism that aims to combine the benefits of human interpretable models with the performance of methods that leverage gradient based optimization. However, the paper does not provide sufficient evidence, empirical or theoretical, to justify the use of these hand-designed neural networks. Some of the major concerns with the paper are:
1. The mechanism is not scalable. Designing individual neurons of a network will likely become intractable for even reasonably complicated tasks.
2. There are no comparisons with existing techniques that aim to learn interpretable policies.
3. The optimization techniques does not seem to provide a substantial performance improvement, and in some cases degrades the performance of the agent.

The lack of improvement via optimization is particularly significant since in the absence of gradient based improvements there seem to be no reason to use this mechanism.
summary_of_the_review: The shortcoming noted in the main review suggest that the proposed mechanism for generating interpretable policies is unlikely to provide a viable alternative to previously proposed methods. Considerable improvements would be required to meet the threshold for acceptance.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: To create an interpretable system, the authors propose to manually construct neural networks such that each neuron corresponds to an interpretable concept. Such a system can then be trained to improve its performance. 
main_review: -The motivation of this work is RL and DL, but the proposed method cannot be used to solve the problems those approach can solve. The connections to RL are explained in an unclear fashion.

-"Trading off" "time spent on training algorithm with the time spent on human design" is at odds with how RL is used. Basic RL can be used without a domain model, human expertise, etc. If a human is available to hand-specify a policy, other approaches are generally preferred. This work is not compared to other approaches for hand-specifying policies.

-After manually specifying parameters, training is performed. It is unclear why this training would preserve the meaning of each neuron.

-The interpretability of the proposed system is not demonstrated.

-The system is changed for the different domains, and different means of explaining it are used. It is unclear how the proposed method would be applied to further domains (since two special cases were presented).

Minor Comments:

-Splitting the Introduction and Related Work would improve clarity.

-This work would benefit from another editing pass.
summary_of_the_review: 1. This work does not present a systematic approach that can be broadly used. 

2. The interpretability of the final model is not shown. 

3. This work attempts to make connections to DRL, but the proposed method is not applicable to the same problems.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposed to design specialized neural networks for different task domains such that the feature space is fully interpretable by human domain experts and therefore offers the opportunity for them to inject semantic biases. 
main_review: This paper proposes an interesting idea of designing specialized neural networks for maximum interpretability, so that the human designer can inject as much semantic information in the feature space as possible to simplify learning. Explainable machine learning is an important topic of research and designing ways to allow humans to inject as much domain knowledge as possible for learning agents is also an important direction for allowing machines to quickly acquire knowledge from their human teachers. However, the method proposed in this paper seems a bit too ad-hoc for specific problems and grants no generalizability to complex real world problems. The two task domains used in this paper are too simple to demonstrate the usefulness of the proposed idea. The writing of this paper is also not very professional and was hard to follow. 

summary_of_the_review: This paper proposes a novel idea for building explainable neural networks but lacks strong theoretical and empirical evidence to support its claim. I do not recommend accepting this paper.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper proposes the Self-Reward Design method for learning a reward function and designing an interpretable neural network for a specific task. It illustrates the method on two gridworld environments, RobotFish and LavaLand, using a hand-designed network for each environment. 
main_review: This paper aims to address the important problem of building intepretable neural networks. Unfortunately, it is poorly written and I found it difficult to follow. 

The general form of the SRD method (independent of the task) is not explained in the paper. The method is illustrated for two environments and seems to be different for each environment, e.g. proposing two different loss functions that both seem ad-hoc and not well-motivated. The explanation of how it works is not clear, e.g. I didn't understand what sentences like this mean: "the prefrontal cortex (PFC) modules judge the worth of the fish’s own action against its environment". The explanations contain a lot of seemingly irrelevant detail, and after reading them I'm still not sure how the SRD method works in general. It would be helpful if the authors could add an algorithm box with some pseudocode for the method.

It seems clear that the proposed method would not scale to more complex environments. The neural networks for the two environments seem to be hand-designed by the authors, and it's not clear how they propose to design a network for a new problem. The paper seems to assume an "imperfect designer" who makes mistakes in the reward function but at the same time is is willing to put a lot of effort into hand-designing a neural network. I think improving the reward function would be much easier for the designer than hand-designing the network as proposed in this paper. 

The paper makes the following strong claim: "The fact that solutions can be achieved by our NN-based models *even without training* means that meaningful and interpretable weights can be purposefully designed to solve problems with more transparency." Since the method was only demonstrated on gridworlds, this claim is not well-supported by the paper.

The paper also claims that the proposed neural network designs are "highly interpretable" but this claim is not well-supported. The experimental section shows various sample trajectories but does not use the interpretable design to explain why the robot follows these trajectories. In fact, the paper suggests that the network weights do not explain the failure modes of the policy: "We see that even failure modes can yield weights profile that look similar to non-failure modes". The collection of experiments is not well-motivated (with unintuitive names like "Compare A") and it was not clear to me how they demonstrate interpretability. 

The paper does not compare the proposed method to any baselines, e.g. any of the related work discussed in the introduction, so the paper does not demonstrate an advantage of the proposed method over other methods. 
summary_of_the_review: This paper is poorly written and the contributions do not seem significant, so I cannot recommend acceptance for this paper.
review_title: null

Meta-review: The paper proposes a design of interpretable neural networks where each neuron is hand-designed to serve a task-specific role, and the network weights can be optimized via a few interactions with the environment. The reviewers acknowledged that the interpretability of neural networks is an important research direction. However, the reviewers pointed out several weaknesses in the paper, and there was a clear consensus that the work is not ready for publication. The reviewers have provided detailed and constructive feedback to the authors. We hope that the authors can incorporate this feedback when preparing future revisions of the paper.
Decision: Reject