Self Reward Design with Fine-grained Interpretability
Authors: Erico Tjoa
Cuntai Guan
Venue: ICLR 2022 Submitted (2022)
Keywords: Reinforcement Learning
Interpretability
Explainable Artificial Intelligence
Neural Networks

Abstract: Transparency and fairness issues in Deep Reinforcement Learning may stem from the black-box nature of deep neural networks used to learn its policy, value functions etc. This paper proposes a way to circumvent the issues through the bottom-up design of neural networks (NN) with detailed interpretability, where each neuron or layer has its own meaning and utility that corresponds to humanly understandable concept. With deliberate design, we show that lavaland problems can be solved using NN model with few parameters. Furthermore, we introduce the Self Reward Design (SRD), inspired by the Inverse Reward Design, so that our interpretable design can (1) solve the problem by pure design (although imperfectly) (2) be optimized via SRD (3) perform avoidance of unknown states by recognizing the inactivations of neurons aggregated as the activation in \(w_{unknown}\).

Under review as a conference paper at ICLR 2022
SELF REWARD DESIGN WITH FINE-GRAINED INTER- PRETABILITY
Anonymous authors Paper under double-blind review
ABSTRACT
Transparency and fairness issues in Deep Reinforcement Learning may stem from the black-box nature of deep neural networks used learn its policy, value fune- tions etc. This paper proposes a way to circumvent the issues through the bottom- up design of neural networks (NN) with detailed interpretability, where each neu- ron layer has its meaning and utility that corresponds to humanly under- standable concept. With deliberate design, we show that lavaland problems can be solved using NN model with few parameters. Furthermore, we introduce the Self Reward Design (SRD), inspired by the Inverse Reward Design, so that our inter- pretable design can (I) solve the problem by pure design (although imperfectly) (2) be optimized via SRD (3) perform avoidance of unknown states by recognizing the inactivations of neurons aggregated the activation in unkkownn
INTRODUCTION
Reinforcement Learning (RL) and Deep Neural Network (DNN) have recently been integrated to solve problems with remarkable performance. The deep reinforcement learning greatly improves the state-of-the-art control and, in the words Sutton & Barto (2018), learning from interaction. Among the well-known successes are (1) the Deep Q-Network (Mnih et al. (2015)) which enabled machine to play Atari Games with incredible performance, and (2) AlphaGo which is capable of playing notoriously complex game of Go (Silver et al. (2016)), which has also been further devel- oped and popularized as being capable of defeating human at pro level. Although DNN has proven interpret. To address this difficulty,
to possess great potentials, it is blackbox that is difficult various works have emerged, thus we have a host of different approaches to eXplainable Artificial Intelligence (XAI); see surveys Arricta et al. (2020); Gilpin et al. (2018); Tjoa & Guan (2020). They have shed some lights into the inner working of DNN, but there may still be large gaps to fill. Note that there no guarantee that interpretability is even attainable, especially when context-dependen interpretability can be subjective.
In this paper, we propose the Self Reward Design, a non-traditional RI. solution that combines highly interpretable human-centric design and the power of DNN. Our robot (representing any artificial agent) rewards itself through purposeful design of DNN architecture, enabling to solve problem without training. The solution might be sub-optimal, but the use of trainable DNN modules (we use pytorch, specifically) addresses the problem. We show that performance improved while interpretability is retained.
This paper is arranged as the following. We start with clarifications. Then we briefly go through re- lated works that inspire this paper. Then our interpretable design and SRD training idea demon- strated with a 1D) toy example, RobotFish. Following that, we introduce SRD on the main robot navigation problem
the 2D lavaland with large focus interpretable design, followed by SRD training and unknown avoidance. Its experimental results will be discussed in the section after, and finally, we conclude with limitations and future works.
THIS PAPER FOCUSES HEAVILY ON INTERPRETABLE HUMAN DESIGN
What exactly is this paper about? Demonstrations how two reinforcement learning problems solved in an interpretable manner through self-reward mechanism. Readers will find how

---

Under review as a conference paper at ICLR 2022
design different components taylored to different problem components. See appendix A.1 for more remarks: the paper has been heavily reorganized to direct readers' focus on interpretable design.
But what is interpretability? While there may be many ways to talk about interpretability, in- terpretability in the context of this paper is fine-grained, i.c. we go all the way down to directly manipulating weights and biases of DNN modules. DNN modules are usually optimized using gra- dient descent from random initialization, thus the resulting weights are hard to interpret. In our SRD model, the meaning and purpose of each neural network component can be explicitly stated with respect to the environmental and model settings.
How do we compare our interpretability with existing explainable deep RL methods? Since we directly manipulate the weights and biases, our interpretability is at a very low level of abstraction, unlike post-hoc analysis c.g. saliency (Greydanus al. (2018)) or semantically meaningful high level specification such as reward decomposition (Juozapaitis et al. (2019)). In other words, we aim to be as transparent as possible, allowing users understand the model from the most basic unit.
Baseline. Our main result on 2D robot lavaland example achieves high performance of approxi- mately 90% accuracy with 10% randomness to allow for exploration. Hence, we believe comparing accuracy performance with other RL. methods is irrelevant, and, most importantly, distracting from our focus of interpretability. If possible, we want to compare the level of interpretability. Quantita- tive comparison tricky, then how do we compare? What baseline to use? We have in fact answered this question in the previous paragraph: our interpretability is fine-grained as we directly manipulate weights and biases.
Different design for different contexts. There is probably no cookie-cutter framework for SRD. Solution design that is interpretable according the above definition necessarily and heavily de- pends on the context, i.e. the shape of the system, thus we might need different formula of losses. This paper demonstrates how purposeful and deliberate designs work on two RI. problems. In our robot fish example, the task is simply to survive, hence the model rewards actions that encourages survival e.g. "eat when hungry". In 2D robot lavaland example, each tile is considered poten- tial checkpoint to reach before reaching the final target, thus the model uses tile-based module that rewards the tile relative to other tiles: the preferred tile is scored higher.
2.1 RELATED WORKS: FROM RL TO DEEP RL TO SRD
RL imperfect human design. RL system can be set up by human manually specifying the rewards. Unfortunately, human design can easily be imperfect since the designers might not necessarily grasp the full extent of complex problem. For RL agents, designers' manual specification of rewards are fallible, subject to errors and problems such as negative side effect of a misspecified reward (Clark & Amoddii (2016)) and reward hacking (Russell & Norvig (2010)). Dylan's inverse reward design (IRD) paper (Hadfield-Mened et al. (2017)) addresses this problem directly: it allows a model to learn beyond what imperfect designers specify. An important component of our solution to imperfect designer problem is the implementation of unknown avoidance, particularly Wynknown in our 2D robot. Also see appendix A.2 for Reward Design Problem (RDP).
From RL to Deep RL. Not only is human design fallible, it might be very difficult, especially for complex problems. In the introduction, we mention that Deep RL (DRL) solves this by combining RL and the power of DNN. However, DRL rewards have been produced by the black-box: they become difficult to understand and thus explainable DRL emerges address that problem.
RL papers that address explaiabbliiyyimeeptrraaittl problems have been compiled in survey papers (eg. Heuillet et al. (2021); Puiutta & Veith (2020)). Saliency, common XAI method, has been applied to visualize deep RL's mechanism (Greydanus et al. (2018)). Relational deep RL uses relational module that not only improves the agent's performance on StarCraft and BBxx-orld, but also provides visualization (2019)). Other methods to improve interpretability include reward decomposition, in which each part of the decomposable reward is semantically meaningful (Juozapaitis et al. (2019)); do refer to the survey papers for several other ingenious designs and investigations into the interpretability of deep RL.
the attention heads useful for interpretability (Zambaldi et al.
From DRL to SRD. Our model is called self reward design because our robot computes its own reward, similar to DRL computation of Q-values. However, human design is necessary put con-
2

---

Under review as a conference paper at ICLR 2022
straints on how self-rewarding is performed that interpretability is maintained. Human designer has the responsibility of understanding the problems, dividing the problems into smaller chunks and then finding the relevant modules to plug into the design in fully interpretable way (see for example how we use convolution layer to create the food location detector, section 2.1).
Computational Efficiency of SRD. The models only use modules from standard DNN modules such as convolution (conv), deconvolution (deconv) and fully-connected (FC) layers with few train- able parameters (e.g. only 180 parameters in Roboi2NN), With proper choice of initial parameters, we can skip the long, arduous training and optimization processes that are usually required for deep learning to learn unseen concepts. We trade off the time spent on training algorithm with the time spent on human design, thus addressing what Kahn et al. (2018) called sample inefficiency (the need for large dataset hence long training time) in a human-centric way.
2.2 OTHER RELEVANT CONCEPTS
Self-supervised Learning. DRL like Value Prediction Network (VPN, Oh et al. (2017)) is self- supervised. Exploration-bass RL algorithm is applied so that data are gathered real-time for train- ing: the model thus optimizes its reward function. Our model is similar in this aspect. Unlike VPN, however, our interpretable design avoids all the abstraction of DNN. Our SRD training is also self- supervised in the sense that we do not require datasets with ground ground-truth labels. Instead, we induce semantic bias via interpretable components to achieve the correct solutions. Also see appendix A.2.
Imagination component. Our pipeline includes several rollouts of possible future trajectories simi- lar to existing RL papers that use imagination components, with differences as the following, Com- pared Kalweit & Boedecker (2017) which optimizes towards target value ÿJ + (heavily abbreviated), SRD (1) is similar because agent updates on every imaginary sample available, but (2) has different, context-dependent loss computation. Also see appendix A.2 for more references.
3 ROBOT FISH: ID TOY EXAMPLE
Problem setting. To broadly illustrate the idea, we use one-dimensional model Fish1 with Fish Neural Network (FishNN) deliberately designed to survive the simple environment. Robot Fish has energy which is represented by a neuron labelled F. Energy diminishes over time. If the energy reaches O, it dies. The environment is enw = [eL, e2, E3] where & = 0.5 indicates there is a food at position i and no food if e; 0. The fish is always located in the first block of env, fig. I(A). In this problem, 'food here' scenario is env = [0.5, 0, 0] which means the food is near the fish. Similarly, 'food there' scenario is env = [0. 0.5, 0 env [0. O, 0.5), which means the food is somewhere ahead and visible. 'No food' scenario is env = 0,,0,0 0j.
e2,
Fish Actions (1) 'eat': recover energy F when there is food in its current position. (2) 'move': movement to the right. In our implementation, 'move' causes env to be rolled left. If we treat the environment as an infinite roll tape and env as fish vision's on the immediately visible blocks, then the food is available every 5 block.
env [0.0.]]
env [0.0,0.5]
env [0.5,0.0]
[0.00.55
= [0.1055
ih
1.0.5]]
Figure 1: Robot Fish setting. The fish is half full/hungry, F 0.5. Left: no food. Middle: food on em1y, thus "food there" neuron lights up. Right: food on envi, thus "food here" neuron lights up.
How to design an interpretable component of neural network? First, we want the fish to be able to distinguish scenarios previously defined: food here food there and no food. Suppose we want neuron that strongly activates when there is food nearby (name it food here neuron, fh), another neuron that strongly activates when there is a food nearby (name it food there neuron, ft), and we want to represent the no-food scenario as 'neither fh and respond'. How do we design a layer with two neurons with the above properties? We use ID convolution layer and selective activation function as (z) e/ [[[[=]]|2 + <), as the following.
3

---

Under review as a conference paper at ICLR 2022
The fh and ft neurons. Define the activation of fh neuron ash oa (con)) where
yo.vys. a ConviD with weight array wih = [1. 0, 0] and bias byn = 0.5. When there is food near the fish, get ffh convolution operator, so dgh = aa asa (Ufn) = 1. This is a strong activation of neuron, because, by design, the maximum value of selective activation function is 1. We not done yet. Similar to a above, define ast. The important task is to make sure that when 'there is food there but NOT HERE', aft activates strongly but ah does not. They are Wft = [0, 1, 1], byt = -0... Together, they form the first layer called the Food Location Detector (FLD). Generally, we have used
(onuy((ene)
[1, 0, 0] * [0.5, 0,0 0.5 0 where denotes the
Orj = 0 sa [conv,
(I)
Interpretable FishNN. To construct the neural network responsible for the fish's actions (eat or move), we need one last step: connecting the neurons plus fish's internal state (energy) (altogether [ay,, Oft, F) to the action output vector leat, move] = [e, m] through FC layer, as shown in fig.
blue dotted box. The FC weights are chosen meaningfully e.g. 'eat when hungry and there is food' and to avoid scenarios like 'eat when there is no food'. This interpretable through manual weight and bias setting.
8
-1) 1
FLD
1
7..5p Sy Convere
7: threhold activation
FC
(1 C
1
Syi softmax
fh
F CC
PFC: pre-frontal costex FC: fully comnecred
ft
env
ft
Conv: convolution
m
F: full
10.4 [blas)
th: food here
ft: food there
SM
rewaard
Figure 2: FishNN architecture, a small value.
Is FishNN's decision correct? The prefrontal cortex (PFC) decides whether FishNN's decision is correct or not. PFC is seen in fig. green dotted box. The name 'PFC' is only borrowed from the neuroscience to reflect our idea that this part of FishNN is associated with internal goals and decisions, similar to real brain (Miller et al. (2002)). How do construct PFC? First, define threshold activation as r(x) Tanh( Leaky Re (x))). Then PFC is constructed deliberately in the same way FishNN is constructed an interpretable way as the following.
First, aggregate states from FishNN into a vector up [ayn, at; F,e, This will be the input PFC. Then, vo is processed using convPFC followed by softmax and threshold activation. The choice of weights and biases can be seen in table 2 in appendix B. With this design, we achieve meaningful activations 01 = [e1, m1, m1,ex] as before. For example, e1 activated when "there is food and the fish eats it when it is hungry", i.e. fh = 1, F < 1 and e is activated relative to mi. The output of PFC binary vector [True, False = [T. Fl obtained from passing V, through a FC layer FCPFC. In this implementation, we have designed the model such that the activation of any V1 neuron is considered a True response; otherwise it is considered false. This is how PFC judges whether FishNN's action decision is correct.
Self reward optimization. As seen above, fish robot has FishNN that decides on an action to take and the PFC that determines whether the action is correct. Is this system already optimal? Yes, if we are only concerned with the fish's 's survival, since the fish will not die from hunger. However, it is not optimal with respect to average energy. We optimize the system through standard DNN backpropagation with the following self reward loss
where CEL is the Cross Entropy Loss and argmaa(22))
loss = CEL(2,
Emeem [T, Fk; is the accumulated decision over mem = 8 iterations to consider past actions (pytorch notation is used). The ground-truth used in the loss is argmaz(2), computed by the fish itself: hence, self-reward design.
Results. Human design ensures survival i.c. problem is solved correctly. Initially, fish robot decides to move rather than eat food when F Rs 0.5, but after SRD training, will prefer to eat whenever
4

---

Under review as a conference paper at ICLR 2022
food is available, shown in fig. 3. New equilibrium is attained: it does not affect the robot's survivability, but fish robot will now survive with higher average energy.
[A]
(B)
w
(C)
105
60-
100
04
0.00
2242
$1750
12780
32770
12700
12140
12093)
Before SRD
Figure 3: (A) plot of energy (F, blue) and food availability (red) for untrained model. SRD-trained model at its early stage looks almost identical. (B) same as (A) but after 12000 SRD training iterations. (C) Average energy of SRD-trained fish model.
4 2D ROBOT IN LAVALAND
The lavaland problem. Like Dylan's IRD paper, we use lavaland as the test bed. Given a map with either dirt (brown), grass (green), lava (red) and target (yellow) tiles and the robot's starting point (blue x), as shown in Fattn of fig 4, the robot chooses to move either UP, DOWN, LEFT or RIGHT reach the target. We treat lava the "unrecognized tile" that the designer forgets take into account (imperfect designer). Problem is considered solved if the target is reached within 36 steps. Robot should prefer not to step on grass tiles but must be designed to avoid lava tile at all cost.
geC_ABA
PFC axsanddiing decision velace
cheee Tanl(y
get wetoorooctake
Mizrget
Vignaes
Digrrsz
Loop til bander taget
Type
Par www.AmoWwyiiion SCHALBIGIT
Rabot 2002
Iileerger
Italase
Figure 4: Robot2NN and PFC. Robot2NN makes 2D robot's action decisions. Fattn (the whole vis- ible map) and Weelf (the agent's position) are the direct input to the model. In U1, Us, red values are positive (desirable), white zero and blue negative (not desirable). 2D robot's 's PFC (green dotted box) resembles RobotFish's PFC in their function of accumulating decision values. (Ugrass]14 [Wgrass]us 23 examples of strong activations that interpretable through tile-based module.
Interpretable design. Like fish robot, we manually choose weights and bias for 2D robot's neural network to solve the problem. Convolutional layers etc are used by Robot2NN to score the tiles (red better, blue worse), i.e. robot creates a favourability gradient. Robot then chooses action that generally moves it from blue red regions.
Notations. ABA: approximate binary array, an array whose entries are expected to be s or 1. DST or DeconvSeq, is the sequence of n deconvolutional layers for a tile type t. Deconvolutional layer, or deconv, a regular DNN module. Each deconv is followed by Tanh activation for normal- ization and non-linearity. Normalization (to magnitude 1) ensures that DeconvSeq compares action choices in relative terms. Tile t denotes the name of a tile, e.g. grass, but it also denotes its [o, 1] normalized RGB value, e.g. for grass, i. [0, 128, 0) /55 array of tile values (they should be

---

Under review as a conference paper at ICLR 2022
obvious from the context). Trecog 10 is the recognition threshold. Designer needs to specify P= {pr :t = target, grass, For now P is the set of untrainable parameters, each ps roughly acting the factor for scaling the true reward, inevitably affecting the robot's final preferences for or against different tiles. They induce the biases that designers input into the model in a simple, interpretable way. We also define the unknown avoidance parameter ua 0.
Interpretable tile-based modules are designed to explicitly map robot's response to each specific tile type. Get ABA( function computes the ABA for each tile type: w; asa pl( [) where pl]] is the mean across RGB channel. This is reminiscent of eq. I; the difference is, each neuron responds to tile type at each atth coordinates. Neuron activations are computed as wn where 1) =target, grass or dirt shown in fig. 4, e.g. strong activation for grass detection occurs at 14
Unknown avoidance. Like Dylan's 's IRD, we have a reliable mechanism for unknown avoidance, the Boolean array Wanknown = [[1 E, > Trecog) to be treated as floating point numbers. From the formula, it can be seen that Wunknown aggregates the negation of known activations. The unknown in our case is the lava tile that the imperfect human designer 'forgets' to account for.
Interpretable tile scores. The score will be used to decide what actions robot will take. It is computed as the following. First, we compute V1. whose components are [ei]c- as the following. For target tiles and "self" tiles (original positions), [ei]e pr DSD [we]- For any other named tiles, also create the favourability gradients based on the initial and target positions, so that, for example, grass tile nearer the target can be valued more than dirt tile. Thus,
[01|e- = p.DS (1Waarget -
* w/]
(3)
Each deconv in DeconvSeq consists of input and output channel. It fully interpretable through our manual selection of kernel weights: kernel has size 3 with center value and side value 0.1 as seen in fig. 5(A). Fig. 5(B, C) show examples of trained weights. This choice of values is intended roughly to create a centred area effect, where the center of the tile contributes most significantly to Wy; for example [01 [ti]taree in fig. 4. Assign es Σg [01]6 this value will be dynamically changed throughout each plan-making process.
(A)
Figure 5: (A) Initial parameters; all deconvs in all DeconvSeq are initialized to the same 3x3 weights with center max value of 1 and 0.1 elsewhere. (B) Once trained, variations of weights are observed for Project A expt 1 (all 180 parameters are shown). (C) Similar to (B) but for project Compare A.
Making one plan. Each plan is series of action, and each action is chosen the following. Set the specific values for the target tile and tile of the original position to Up and -up respectively, where to max { [|z | is computed without backpropagation gradient to prevent any complica- tions. Finally, to incorporate lava avoidance, or generally avoidance of anything previously unseen, es <- uy (1 Wanknown) + * Wunknown. From the current position, robot's 's neighbouring [vz]; where i = up, down, left, right values are collected, and neighbours with the top two values chosen. From the top two choices, randomly choose one of them with the action with higher value. The randomness is to encourage exploration. After each action, the tile the robot leaves will be assigned 0.9vn prevent the robot from oscillating back and forth, where vo is separately computed inside step_update() function. A series of actions are chosen in this manner until the target is reached or a maximum of 36 iterations are reached. Thus we have obtained a plan and its score Uptan N Di.jeg [Ux]i,j. the mean of all values assigned to the chosen tiles where Ç is the trajectory (see fig. orange arrows in Uy:).
odds, favouring
Imagining multiple plans for SRD optimization. Robot makes plans by imagining Nplans = different trajectories to reach the target. It plans and executes the plan with the highest Uplan- Like

---

Under review as a conference paper at ICLR 2022
Kalweit & Boedecker (2017), we use all Rplans imagination branches for training (SRD optimiza- tion) with possibly novel loss minimization:
loss = Enipeee (1 (an (NUpaan,)))
(4)
where [i] normalizes Uplan to the magnitude of I, where normalization factor is computed without gradient to prevent complication. Thus, robot's PFC always undervalues its reward relative to an abstract, ideal value 1. The intended effect is to always try to maximize Uplan. A standard Stochastic Gradient Descent with learning rate 10-4 used for optimization. This resembles fish's PFC true or false neurons used for rewarding itself, except it is continuous.
EXPERIMENT AND RESULTS
Experimental setup and comparison. We experiment on several initial settings as the following. For each experiment, we randomly generate and save 4096 maps to evaluate the performance of both standard design and SRD trained model, and similarly 4096 maps for SRD training. Note: All codes are included, including codes for creating animated gifs of robot traversing the lavaland. See appendix C.I regarding multiple trials of the same experiments we conducted for reproducibility (project B, C, D etc).
Reaching targets without training, optimized by training. Table shows that even without train- ing ("No SRD" columns), our interpretable designs have enabled relatively high rate of problem solving. With SRD, the accuracies further improved.
Table I: Accuracy comparisons for Project A, project Compare A, project With-Lava-A and project Lava NOAV A. Acc Rreached target/ torget/4096. /099.. "No SRD" indicates no training: reasonable accuracy
is attainable purely with design. SRD training generally increases the accuracies. Starred values reflect failure modes, matching their respective histograms; see fig. and
Project A
Compare A
With Lava A
Lava NOAV A
Expt no.
No SRD
SRD
No SRD
SRD
No SRD
SRD
No SRD
SRD
0.767
0.856
0.862
0.941
0.826
0.893
0.841
0.884
2
0.765
0.845
0.871
0.589*
0,821
0.893
0.850
0.909
3
0.763
0.817
0.874
0.930
0,819
0.837
0.844
0.868
4
0.760
0.868
0.863
0.422*
0,823
0.762
0.835
0.907
Project A: standard design and SRD on maps without lava tiles. The fraction of grass tiles is 0.3. No robot will be spawned the target immediately. Here, we use Pdirt.grass 0.2, -0.8 chosen empirically. For training, we only run through epoch, taking a very short time to complete (less than 0.5 hour per project without GPU). Fig. 6(B) shows the cumulative histogram of the number of tiles traversed with Ptarget 2.
Compare A: model is more focused on the target. This setting is similar Project A, but we use more extreme parameter setting, i.e. set Ptarget = 10. The local reward of the target tile is increased, and from table 1, we see that this generally increases the accuracies of both standard and SRD trained model (more likely to reach the target tile). The value seems to produce more unstable results as well, i.e. we empirically observe more failure modes.
Project Lava A. In this experiment, 0.1 of the tiles randomly assigned as lava and unknown avoid- ance u, = 2. We also set Ptarget = thus, similar to Compare A we see higher accuracy but some failure modes as well. The model is designed to not recognize lava tile, so a lava tile will activate the robot's 's Robot2NN Wunknown at lavas' positions. Fig. 7(A) is just an example of how robot successfully avoids lava, while fig. 7(C) lava cumulative histograms show that there are nearly zero lava tiles traversed.
Project Lava NOAV A. Project Lava NOAV A is similar Project Lava A, but set ua = 0, i.e. no unknown avoidance. The results are clear, we see in 7(D) that robot will traverse the lava tiles without much regards.
Robot2NN weights and preserved interpretability. This model is very efficient because it consists of only 180 trainable parameters, shown fully in fig. 5(B,C). As expected of relatively simple

---

Under review as a conference paper at ICLR 2022
(8.
(8)
4000
2030
33
30
20
30
64
(1)
standadd
Talanaod
2000
SLL
4200
$
10
30
90
0
20
30
20
30
20
30
CX07
CKg? 3
Expt
Figure 6: (A) A sample trajectory from Project A. (B) Cumulative histogram of the no. of tiles traversed by robot in Project A. Steeper histogram indicates that less tiles of the type are being traversed (for grass, steeper better) (C) sample trajectory from Compare (D) Similar B, but for robot in Compare A.
a0ce
od
3000
(6)
Noah dosee
Me Mrlocondla cim tornne
200+
2000
Figure 7: (A) sample trajectory from Lava A. (B) A sample trajectory from Lava NOAV A. A lava tile is traversed. (C) Cumulative histogram of the no. of tiles traversed by robot in Lava A. Almost zero lava tiles are traversed. (D) Similar to (B) but for Lava NOAV A, which some lava tiles are traversed.
problems, there no need for millions of parameters required to achieve high accuracy. High performance 90% accuracy is attained, given 10% randomness is allowed. The weights of target deconv appear to have been trained towards higher positive values (redder). The center value remains the most prominent for all, thus preserving interpretability. Looking into individual variations, fig. 5(B) shows the weights from Robot2NN model of project A expt while fig. 5(C) from project Compare A expt 1. The difference in grass weights apparent. See appendix C.2 on (1) preserved interpretability (2) why no conclusion should be drawn from this observation.
CONCLUSION, LIMITATION AND FUTURE DIRECTIONS
Generalizability and scalability. An important limitation to this design is the possible difficulty in creating specific design for very complex problems, for example, computer vision problems with high-dimensional state space. Problems with multitude of unknown variables might be difficult to factor into the system, or, if they are factored in, the designers' imperfect understanding the vari- ables may create poor model. Future works applying interpretable SRD complex systems might push Deep RL community into designing systems that aim not only to attain high perfor- mance, but also be as highly interpretable our design. Controlling Weights and Biases. A
8

---

Under review as a conference paper at ICLR 2022
future study regularizing weights may be interesting. While have shown that design provides a good consistency between the interpretable weights before and after training. it is not surprising that the combination of small differences in weights can yield different final weights that still perform well. So far, is not clear how different parameters affect the performance of a model, or if there are any ways to regularize the training of weight patterns towards something more inter- pretable and consistent. So far, there is no guarantee for convergence either, and it may be difficult to provide a general proof, since the design heavily dependent on the context. A technical limita- tion is the lack of APIs that perform the exact operations we may need to parallelize the imaginative portions of SRD optimization.
Conclusion. Finally, we have demonstrated interpretable neural network design with fine- grained interpretability. Human experts purposefully design models that solve specific problems based on their knowledge of the problems. SRD is introduced as way improve performance on top of the designers' imperfection. The system highly interpretable with manual adjustment of weights and biases. It is also efficient since very few weights are needed compared to traditional DNN.
ETHICS STATEMENTS
There is no specific ethical statement present. We demonstrate methods for designing neural net- works for RL problem in transparent way. We believe each part is transparently presented, and, the paper is insufficient, the code in supp. materials should fully complements each description made in this paper.
REPRODUCIBILITY STATEMENTS
All codes are available in the supp. materials, to be released to public repository in case of accep- tance. Results are tested different random number seeds, and irregularities mode collapses have been presented clearly in the paper as well (see for example the annotations made on the cumulative histograms).
ACKNOWLEDGMENTS
Anonymous now
REFERENCES
Alejandro Barredo Arrieta, Natalia Rodriggdd, Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador Garcia, Sergio Gil-Lopez, Daniel Molina, Richard Benjamins, Raja Chatila, and Francisco Herrera. Explainable artificial intelligence (xai): Concepts, tax-
onomies, opportunities and challenges toward responsible ai. Information Fusion, 58:82
115, 2020. ISSN 1566-2535. doi: https://doi.org/10.. 10f.///...nffoss2201120000111110 URL http: / //www. www.scciennedireet.. www.cooo//cc/ooo http://ccceceee.aaaaaaaa/////aa/
Jack Clark and Dario Amodei. Faulty reward functions in the wild, 2016. URL https:
//openai. com/bl http//llll//ao// faw.falllllyaddddeeaaaaaa rewa--eeaad---o Functions: Accessed: 2021-08-21.
L. H. Gilpin, D. Bau, B. Z. Yuan, A. Bajwa, M. Specter, and L. Kagal. Explaining explanations: An
overview of interpretability of machine learning. In 2018 IEEE 5th International Conference on Data Science and Advanced Analytics (DSAA), pp. 80-89, 2018.
Samuel Greydamus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding Atari agents. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 1792-1801. PMLR, 10-15 Jul 2018. greydanus 8a html.
URL https: htttht:///pppoceeeeee http///ooeediicccccocoooo ngs. ml press/v8C.
Dylan Hadfield-Menell. Smitha Milli, Pieter Abbeel, Stuart Russell, and Anca D. Dragan. Inverse reward design. In Proceedings of the 31st International Conference on Neural Information Pro- cessing Systems, NIPS' 17, pp. 6768-6777 Red Hook, NY, USA, 2017. Curran Associates Inc.
ISBN 18860964.

---

Under review as a conference paper at ICLR 2022
Danijar Hafner, Timothy Lillicrap. Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. In International Conference on Learning Representations, 2020.
URL
Alexandre Heuillet, Fabien Couthouis, and Natalia Díaz-Rodriguez. Explainability in deep
reinforcement learning. Knowledge-Basee Systems, 214:106685, 2021. ISSN 0950-7051.
doi: https://do..or//100 111.11011........002220001. 200.0000...2220.0088.... 106685. URL /wt.Lttt/ww...sieeeeeeee. http://www.iinneeiee......
com/ wwwpnn/eeeriiieeeeeeeeo http:////oaaaaaaaoo/o///////////
Z. Juozapaitis, A. Koul, A. Fern, M. Erwig, and F. Doshi-Velez. Explainable reinforcement learning via reward decomposition. In in proceedings at the International Joint Conference Artificial
Intelligence. A Workshop on Explainable Artificial Intelligence. 2019.
Gregory Kahn, Adam Villaflor, Bosen Ding, Pieter Abbeel, and Sergey Levine. Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 5129-5136, 2018. doi: 10. 10.1109// 8.6666555..
Gabriel Kalweit and Joschka Boedecker. Uncertainty-driver imagination for continuous deep rein- forcement learning. In Sergey Levine, Vincent Vanhoucke, and Ken Goldberg (eds.), Proceedings of the 1st Annual Conference Robot Learning, volume 78 of Proceedings Machine Learn- ing Research, pp. 195-206. PMLR, 13-15 Nov 2017. URL https: : http://:///ccceingss../.. ammiinggsm........ mmlr. press/ peessvvii / wwa.walwwlaaaaaaamammmamaaaaalll html.
Earl K. Miller, David J. Freedman, and Jonathan D. Wallis. The prefrontal cortex: categories, concepts and cognition. Philosophical transactions of the Royal Society of London. Series B, sciences. sciencss 357(1424) Aug 2002. ISSN 0962- doi: 10. 10.000./ 1099. URL https: http///wweeeeeeee////// wwccciinnnng wtw.:iin.......oooo.o.........mm wtt.mm.n................
12217179(pmid].
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin Riedmiller, Andreas K. Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Hclen King, Dharshan Kumaran, Daan Wier- stra, Shane Legg. and Demis Hassabis. Human-level control through deep reinforcement learn- ing. Nature, 518(7540))552 -33, Feb 2015. ISSN 1476-4687. doi: 10. 10.10//atuure142366 URL https:://o// org/ 1038 0038/naturee hatuure4236....
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL #fbd6cbb01 9a1 0111111310001229930001111
pdf.
Erika Puiutta and Eric M. S. P. Veith. Explainable reinforcement learning: A survey. In Andreas Holzinger, Peter Kieseberg, A Min Tjoa, and Edgar Weippl (eds.), Machine Learning and Knowl- edge Extraction, pp. 77-95, Cham, 2020. Springer International Publishing. ISBN 978-3-030- 57321-8.
Sébastien Racanière, Théophane Weber, David Reichert, Lars Buesing, Arthur Guez, Danilo Rezende, Adria Puigdomènech Badia, Oriol Vinyals, Nicolas Heess, Yujia Li, Razvan Pascanu, agents
Peter Battaglia, Demis Hassabis, David Silver, and Daan Wierstra. Imagination-augmentt for deep reinforcement learning. In Proceedings of the 31 International Conference on Neural Information Processing Systems, NIPS' 17, pp- 5694-5705, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
Stuart Russell and Peter Norvig. Artificial Intelligence: A Modern Approach. Prentice Hall, edition, 2010.
David Silver, Aja Huang, Chris J. Maddison, Arthur Guez, Laurent Sifre, George van den Driessche, Julian Schrittwieser, Ioannis Antonoglou, Veda Panneershelvam, Marc Lanctot, Sander Dieleman, Dominik Grewe, John Nham, Nal Kalchbrenner, Ilya Sutskever, Timothy Lillicrap, Madeleine Leach, Koray Kavukcuoglu, Thore Graepel, and Demis Hassabis. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, Jan 2016. ISSN 1476-4687. doi: 10. no.tt 16961. URL https: /oii /10 1038 hwt./eatuureeeeeeeeeeee 6961.
10

---

Under review as a conference paper at ICLR 2022
Satinder Singh, A. Barto, and Richard L. Lewis. Where do rewards come from. 2009.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning introduction. 2018.
Erico Tjoa and Cuntai Guan. A survey on explainable artificial intelligence (xai): Toward medical xai. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-21, 2020. doi: 10.1 109/ TNNLS 2020.3027314
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter Battaglia. Deep re- inforcement learning with relational inductive biases. In International Conference on Learning Representations, 2019. URL https::///// www..eeee..ee.eeeeoeeeeeeeeeewee net forum? F=HKKaF0C9K

---

Under review as a conference paper at ICLR 2022
A APPENDIX
A.1 THIS PAPER FOCUSES HEAVILY ON INTERPRETABLE HUMAN DESIGN
We reorganize this paper heavily based on ICLR 2022 reviewers' comments. Thanks to them, we understand that our focus on fine fine-grained interpretability (through direct manipulation of weight and biases) seems to have been lost or easily overlooked the previous version of our paper. There is a tendency to focus on performance and solution to generalizability. We have answers for this (1) we do achieve high performance, and comparison might be very redundant. Our intention is to compare interpretability instead of performance. (2) Scalability. We naturally start by demonstrating the design on simple problems. Starting with a highly complex problems may be counterproductive and very difficult to present. In this revised paper, we have focused the clarity of presentation.
A.2 RELATED WORKS
In Reward Design Problem, Singh et al. (2009) observes that, given bounded agents, proxy reward function can be more optimal than the true fitness function which distinct from the proxy. Dylan's IRD, on the other hand, approximates the true fitness function given an observed proxy reward function, assuming the designer is the bounded agent, i.e, the designer is fallible. IRD performs inversion to compute P(w = w" V. M) from P(W)w*, .M). where w is the weight of the reward function r(;; w), where is the trajectory. We cannot fit the description of our model exactly in the same language, since we opt for an interpretable design with non- con-traditional RL reward. The closest have to reward is Uplan in Robot2NN, shown in fig. 4 indirectly as the values summed through orange arrows in by described in the next paragraph.
Comparison with standard Deep RL. In the main text, mention that traditional reward function. Deep Q-learning loss function (0,) computes differences between current Q value (value of the current state-action pair) and its temporal difference (TD)² given by r r+ ymazz Q(s' ss", aa, (;))).. The reward can be fixed by the designer; this can be seen on OpenAl's official github", and the values are then used to update Q. an expected value of total future rewards, technically also a reward. SRD is similar to DQN in this sense: there is future cumulative reward. In turn, DQN similar to standard RL except Q predicted by the neural network. However, SRD does not apply the loss function L, (0.) in DQN since we do not specify TD, which is dictated by the reward that explicitly adds into the sum of future reward. As mentioned, the reward value r in DQN is specified by the designer and thus DQN robot aims to maximize values whose absolute magnitude of worth is known. By contrast, SRD lets the robot measure the worth of every series of actions. Since the target Tanh score is 1, every measurement of worth is relative with respect to the learning process. Even then, when the robot compares the multiple plans it makes at once, it has its own yardstick for rewarding its own decisions. The absolute magnitude of reward based on this yardstick is decided by the robot's own neural network, thus the "self reward". Finally, our robot self reward works by always undervaluing its perceived reward w.r.t the ideal plan with score 1, mentioned in the main text.
use non-
The following supplements the main text section 2.2.
Self-supervision. Kahn et al. (2018) demonstrates self-supervised DRL on a simulated car that learns from real-time experience; the paper also applies the model real-world RC car. After hybrid graph-based RL model is incorporated. it travels around avoiding collisions without human intervention. Our robots self-supervise similarly. After the initial interpretable design, 2D Robot moves towards the target, favoring or avoiding different types of tiles without human supervision. In particular, grass tiles are recognizable, but less desirable, while lava tiles are "not recognized", hence must be completely avoided. Informally, the robots have their own preferences on how to assign the eventual values of the actions with known local consequence, while they are careful about their own ignorance induced by imperfect design (unknown avoidance).
Imagination components. Unlike the rollout in Racanière et al. (2017), cach of our SRD rollout consists of series of asymmetric binary choices a, d2 chosen from {a E A) so that v(a,) 2 v(a2),
'see for example https: wwwsor fl.. http:///gggeaaaaooorraaaaa////// Phtpps:/ / /gi thub. com ooenaii http:///w..aategg//////////aasaa cartpole.py
12

---

Under review as a conference paper at ICLR 2022
where A is the set of actions and is any generic function that gives each action local value. The values will be aggregated into a self-reward. Dreamer (Hafner et al. (2020)) solves RL problem using only latent imagination where many models (such as reward, action and value models) specified as probability distributions. By contrast, SRD creates no specific sub-model. All values just NN activations, and they will be aggregated into impromptu, just-in-time scores, based on which the plans can be greedily chosen.
B
ROBOT FISH
Table 2: Weights and biases in Robot Fish design in main text section 3.
Output
Conv weights
bias
f
1,0,0
0.55
aft
0, 1,
0.5
0, 11,0,-1,1,0
None
m1
00,1,
None
er
[-1,-1,0,0,1]
None
C 2D ROBOT
C.I MULTIPLE TRIALS OF THE SAME EXPERIMENTS
We perform multiple trials of each experiment for reproducibility and consistency. The main text only reports the first trial for each experimental setting since the result is generally representative. Each trial also consists of 4 experiments. Other results are in the supplementary materials (full version will be released later).
Repeat trials for Project A: project B, and D. Histogram for project can be seen at fig.
Repeat trials for Compare A: compare B, C and D.
Repeat trials for Lava Lava B, C, D.
Repeat trials for Lava NOAV A: Lava NOAV B, C, D.
standard
srd
2000
4000
2000
30
10
20
10
30
expt
expt
expt
Figure 8: Project cumulative histograms.
C.2 DECONV WEIGHTS
(1) Preserved interpretability. We see from fig. C.2 that the trained models still have the general interpretable shape initiated it with. While there is theoretical proof, there may be intuitive reason. Due to our interpretable design, the model starts off with reasonable ability to solve the problem. This probably means weights and biases already reside in high dimensional parameter
13

---

Under review as a conference paper at ICLR 2022
space somewhere around one of the local minima, and this local minimum is interesting because more interpretable i.. weights have recognizable shapes as we have initiated. As result, short training leads it nearer that local minimum, hence the overall shape of the model remains similar to the initial shape, thus interpretable.
(2) Why conclusion should be drawn from this observation? It seems that project A with smaller Prarger results in relatively less preference for the grass tiles, which in turn leads to negative values (blue) for deconv for grass tiles. By comparison, project Compare negative values for grass tiles deconv. Unfortunately, the weights are shown only for demonstration: there is no definite conclusion that can be drawn. This is because other experiments similar to project A also can result in all positive deconv weights with different patterns. They still yield high aceuracy, thus possible variations within even this small set of parameters can still produce similar performance. Other results are shown in the appendix. Lava A project does not yield particularly distinct patterns. We see that even failure modes can yield weights profile that look similar non- failure modes. Further investigations may be necessary
seems have no
Figure 9: Weights for (A) Project Lava expt (B) Project Lava NOAV expt (C) Lava A expt
4.
(B)
decom
deconu
deconv
decomv
Figure 10: Weights for failure modes (A) Project Compare A expt (B) Project Compare expt 4. (A) still shows standard-looking weights.
14


Reviews:
[{"review_id": "ygtNWccAuI", "reviewer": "Reviewer_NiCV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Interpretability for deep reinforcement learning is an important and significant topic of research. This paper suggests a mechanism that aims to combine the benefits of human interpretable models with the performance of methods that leverage gradient based optimization. However, the paper does not provide sufficient evidence, empirical or theoretical, to justify the use of these hand-designed neural networks. Some of the major concerns with the paper are:\n1. The mechanism is not scalable. Designing individual neurons of a network will likely become intractable for even reasonably complicated tasks.\n2. There are no comparisons with existing techniques that aim to learn interpretable policies.\n3. The optimization techniques does not seem to provide a substantial performance improvement, and in some cases degrades the performance of the agent.\n\nThe lack of improvement via optimization is particularly significant since in the absence of gradient based improvements there seem to be no reason to use this mechanism.", "summary_of_the_paper": "This paper proposes using individually designed, interpretable neural networks to solve a given task. Each neuron in the network is hand-designed to serve a specific task-dependent purpose. These hand-designed solutions are then optimized using environment interactions. The neuron level design of the network is used to provide human interpretability, as the behavior of the network can be understood based on the activations during environment interactions. The neural network structure of the policy can then be optimized via gradient descent to provide a performance improvement over the hand-designed solution.", "summary_of_the_review": "The shortcoming noted in the main review suggest that the proposed mechanism for generating interpretable policies is unlikely to provide a viable alternative to previously proposed methods. Considerable improvements would be required to meet the threshold for acceptance.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "gesEyVLmdbM", "reviewer": "Reviewer_sPTc", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "-The motivation of this work is RL and DL, but the proposed method cannot be used to solve the problems those approach can solve. The connections to RL are explained in an unclear fashion.\n\n-\"Trading off\" \"time spent on training algorithm with the time spent on human design\" is at odds with how RL is used. Basic RL can be used without a domain model, human expertise, etc. If a human is available to hand-specify a policy, other approaches are generally preferred. This work is not compared to other approaches for hand-specifying policies.\n\n-After manually specifying parameters, training is performed. It is unclear why this training would preserve the meaning of each neuron.\n\n-The interpretability of the proposed system is not demonstrated.\n\n-The system is changed for the different domains, and different means of explaining it are used. It is unclear how the proposed method would be applied to further domains (since two special cases were presented).\n\nMinor Comments:\n\n-Splitting the Introduction and Related Work would improve clarity.\n\n-This work would benefit from another editing pass.", "summary_of_the_paper": "To create an interpretable system, the authors propose to manually construct neural networks such that each neuron corresponds to an interpretable concept. Such a system can then be trained to improve its performance. ", "summary_of_the_review": "1. This work does not present a systematic approach that can be broadly used. \n\n2. The interpretability of the final model is not shown. \n\n3. This work attempts to make connections to DRL, but the proposed method is not applicable to the same problems.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "STxVFU25lxl", "reviewer": "Reviewer_NCWP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes an interesting idea of designing specialized neural networks for maximum interpretability, so that the human designer can inject as much semantic information in the feature space as possible to simplify learning. Explainable machine learning is an important topic of research and designing ways to allow humans to inject as much domain knowledge as possible for learning agents is also an important direction for allowing machines to quickly acquire knowledge from their human teachers. However, the method proposed in this paper seems a bit too ad-hoc for specific problems and grants no generalizability to complex real world problems. The two task domains used in this paper are too simple to demonstrate the usefulness of the proposed idea. The writing of this paper is also not very professional and was hard to follow. \n", "summary_of_the_paper": "This paper proposed to design specialized neural networks for different task domains such that the feature space is fully interpretable by human domain experts and therefore offers the opportunity for them to inject semantic biases. ", "summary_of_the_review": "This paper proposes a novel idea for building explainable neural networks but lacks strong theoretical and empirical evidence to support its claim. I do not recommend accepting this paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "QrNHlk-W3dU", "reviewer": "Reviewer_cYQC", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper aims to address the important problem of building intepretable neural networks. Unfortunately, it is poorly written and I found it difficult to follow. \n\nThe general form of the SRD method (independent of the task) is not explained in the paper. The method is illustrated for two environments and seems to be different for each environment, e.g. proposing two different loss functions that both seem ad-hoc and not well-motivated. The explanation of how it works is not clear, e.g. I didn't understand what sentences like this mean: \"the prefrontal cortex (PFC) modules judge the worth of the fish’s own action against its environment\". The explanations contain a lot of seemingly irrelevant detail, and after reading them I'm still not sure how the SRD method works in general. It would be helpful if the authors could add an algorithm box with some pseudocode for the method.\n\nIt seems clear that the proposed method would not scale to more complex environments. The neural networks for the two environments seem to be hand-designed by the authors, and it's not clear how they propose to design a network for a new problem. The paper seems to assume an \"imperfect designer\" who makes mistakes in the reward function but at the same time is is willing to put a lot of effort into hand-designing a neural network. I think improving the reward function would be much easier for the designer than hand-designing the network as proposed in this paper. \n\nThe paper makes the following strong claim: \"The fact that solutions can be achieved by our NN-based models *even without training* means that meaningful and interpretable weights can be purposefully designed to solve problems with more transparency.\" Since the method was only demonstrated on gridworlds, this claim is not well-supported by the paper.\n\nThe paper also claims that the proposed neural network designs are \"highly interpretable\" but this claim is not well-supported. The experimental section shows various sample trajectories but does not use the interpretable design to explain why the robot follows these trajectories. In fact, the paper suggests that the network weights do not explain the failure modes of the policy: \"We see that even failure modes can yield weights profile that look similar to non-failure modes\". The collection of experiments is not well-motivated (with unintuitive names like \"Compare A\") and it was not clear to me how they demonstrate interpretability. \n\nThe paper does not compare the proposed method to any baselines, e.g. any of the related work discussed in the introduction, so the paper does not demonstrate an advantage of the proposed method over other methods. ", "summary_of_the_paper": "The paper proposes the Self-Reward Design method for learning a reward function and designing an interpretable neural network for a specific task. It illustrates the method on two gridworld environments, RobotFish and LavaLand, using a hand-designed network for each environment. ", "summary_of_the_review": "This paper is poorly written and the contributions do not seem significant, so I cannot recommend acceptance for this paper. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IoKU1Y1_wJJ", "reviewer": "Reviewer_cBTu", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Throughout the paper, the only thing that makes sense in my opinion is that the goal of incorporating human knowledge into the network design. \n\nWeaknesses:\n- Terrible writing. Really hard for readers to understand. Lots of unexplained notations and technical details but much less illustration of the intuition.\n- Unreasonable technical approach. Why does the network need to be trained if the goal is just mimicking the greedy action? Can we just use the greedy output instead? \n- Hard to generalize. The network is designed case by case with heavy human effort, I don't see any learnable component in the network that makes sense and effectively supplements the human design.\n- Unconvincing results. This is probably because the technical approach is not reasonable. SRD fails to improve the performance in general by looking at the result figures.", "summary_of_the_paper": "This paper proposes incorporating (a large amount of) human knowledge into policy network design such that it solves the problem directly or within a few iterations of training. The network is trained by the proposed Self Reward Design (SRD) mechanism to simply consolidate its greedy decision, which as claimed by the authors would help supplement the human design of the network.", "summary_of_the_review": "A clear rejection based on the weaknesses mentioned above and no intuition to the community.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a design of interpretable neural networks where each neuron is hand-designed to serve a task-specific role, and the network weights can be optimized via a few interactions with the environment. The reviewers acknowledged that the interpretability of neural networks is an important research direction. However, the reviewers pointed out several weaknesses in the paper, and there was a clear consensus that the work is not ready for publication. The reviewers have provided detailed and constructive feedback to the authors. We hope that the authors can incorporate this feedback when preparing future revisions of the paper.
Decision: Reject