Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposed a novel framework Neuron as an Agent (NaaA) for training neural networks to perform various machine learning tasks, including classification (supervised learning) and sequential decision making (reinforcement learning). The NaaA framework is based on the idea of treating all neural network units as self-interested agents and optimizes the neural network as a multi-agent RL problem. This paper also proposes adaptive dropconnect, which extends dropconnect (Wan et al., 2013) by using an adaptive algorithm for masking network topology.
 
This work attempts to bring several fundamental principles in game theory to solve neural network optimization problems in deep learning. Although the ideas are interest and technically sound, and the proposed algorithms are demonstrated to outperform several baselines in various machine learning tasks, there several major problems with this paper, including lacking clarity of presentation, insights and substantiations of many claims. These issues may need a significant amount of effort to fix as I will elaborate more below.
 
1. Introduction
There are several important concepts, such as reward distribution, credit assignment, which are used (from the very beginning of the paper) without explanation until the final part of the paper.
 
The motivation of the work is not very clear. There seems to be a gap between the first paragraph and the second paragraph. The authors mentioned that “From a micro perspective, the abstraction capability of each unit contribute to the return of the entire system. Therefore, we address the following questions. Will reinforcement learning work even if we consider each unit as an autonomous agent ”
Is there any citation for the claim “From a micro perspective, the abstraction capability of each unit contribute to the return of the entire system” ?  It seems to me this is a very general claim. Even RL methods with linear function approximations use abstractions.  Also, it is unclear to me why this is an interest question. Does it have anything to do with existing issues in DRL? Moreover, The definition of autonomous agent is not clear, do you mean learning agent or policy execution agent?
 
“it uses \epsilon-greedy as a policy, …” Do you mean exploration policy?
I also have some concerns regarding the claim that “We confirm that optimization with the framework of NaaA leads to better performance of RL”. Since there are only two baselines are compared to the proposed method, this claim seems too general to be true.
 
It is not clear to why the authors mention that “negative result that the return decreases if we naively consider units as agents”. What is the big picture behind this claim?
 
“The counterfactual return is that by which we extend reward …” need to be rewritten.
 
The last paragraph of introduction discussed the possible applications of the proposed methods without any substantiation, especially neither citations nor any related experiments of the authors are provided.
 
2 Related Work
 
“POSG, a class of reinforcement learning with multiple ..” -> reinforcement learning framework
 
“Another one is credit assignment. Instead of reward.. ” Two sentences are disconnected and need to be rewritten.
 
“This paper unifies both issues” sounds very weird. Do you mean “solves/considers both issues in a principled way”?
 
The introduction of GAN is very abrupt. Rather than starting from introducing those new concepts directly, it might be better to mention that the proposed method is related to many important concepts in game theory and GANs.
 
“,which we propose in a later part of this paper” -> which we propose in this paper
 
 
3. Background
 
“a function from the state and the action of an agent to the real value” -> a reward function  
 
Should provide a citation for DRQN
 
There is a big gap between the last two paragraphs of section 3.
 
4. Neuro as an agent
 
“We add the following assumption for characteristics of the v_i” -> assumptions for characterizing v_i
 
“to maximize toward maximizing its own return” -> to maximize its own return
 
We construct the framework of NaaA from the assumptions -> from these assumptions
 
“indicates that the unit gives additional value to the obtained data. …” I am not sure what this sentence means, given that \rho_ijt is not clearly defined.
 
5. Optimization
 
“NaaA assumes that all agents are not cooperative but selfish” Why? Is there any justification for such a claim?
 
 
What is the relation between \rho_jit and q_it ?
 
“A buyer which cannot receive the activation approximates x_i with …” It is unclear why a buyer need to do so given that it cannot receive the activation anyway.
 
“Q_it maximizing the equation is designated as the optimal price.” Which equation?
 
e_j and 0 are not defined in equation 8
 
 
6 Experiment
setare -> set are
 
what is the std for CartPole in table 1
 
It is hard to judge the significance of the results on the left side of figure 2. It might be better to add errorbars to those curves
 
More description should be provided to explain the reward visualization on the right side of figure 2. What reward? External/internal?
 
“Specifically, it is applicable to various methods as described below …” Related papers should be cited.
review_title: Interesting and promising ideas, but need significant polishing and substantiation 

final_score: rating: 7: Good paper, accept
review_content: In this paper, the authors present a novel way to look at a neural network such that each neuron (node) in the network is an agent working to optimize its reward. The paper shows that by appropriately defining the neuron level reward function, the model can learn a better policy in different tasks. For example, if a classification task is formulated as reinforcement learning where the ultimate reward depends on the batch likelihood, the presented formulation (called Adaptive DropConnect in this context) does better on standard datasets when compared with a strong baseline.

The idea proposed in the paper is quite interesting, but the presentation is severely lacking. In a work that relies heavily on precise mathematical formulation, there are several instances when the details are not addressed leading to ample confusion making it hard to fully comprehend how the idea works. For example, in section 5.1, notations are presented and defined much later or not at all (g_{jit} and d_{it}). Many equations were unclear to me for similar reasons to the point I decided to only skim those parts. Even the definition of external vs. internal environment (section 4) was unclear which is used a few times later. Like, what does it mean when we say, “environment that the multi-agent system itself touches”?

Overall, I think the idea presented in the paper has merit, but without a thorough rewriting of the mathematical sections, it is difficult to fully comprehend its potential and applications.
review_title: Good work. Presentation needs further polishing.

final_score: rating: 3: Clear rejection
review_content: The authors consider a Neural Network where the neurons are treated as rational agents. In this model, the neurons must pay to observe the activation of neurons upstream. Thus, each individual neuron seeks to maximize the sum of payments it receives from other neurons minus the cost for observing the activations of other neurons (plus an external reward for success at the task).  

While this is an interesting idea on its surface, the paper suffers from many problems in clarity, motivation, and technical presentation. It would require very major editing to be fit for publication. 

The major problem with this paper is its clarity. See detailed comments below for problems just in the introduction. More generally, the paper is riddled with non sequiturs. The related work section mentions Generative Adversarial Nets. As far as I can tell, this paper has nothing to do with GANs. The Background section introduces notation for POMDPs, never to be used again in the entirety of the paper, before launching into a paragraph about apoptosis in glial cells. 

There is also a general lack of attention to detail. For example, the entire network receives an external reward (R_t^{ex}), presumably for its performance on some task. This reward is dispersed to the the individual agents who receive individual external rewards (R_{it}^{ex}). It is never explained how this reward is allocated even in the authors’ own experiments. The authors state that all units playing NOOP is an equilibrium. While this is certainly believable/expected, such a result would depend on the external rewards R_{it}^{ex}, the observation costs \sigma_{jit}, and the network topology. None of this is discussed. The authors discuss Pareto optimality without ever formally describing what multi-objective function defines this supposed Pareto boundary. This is pervasive throughout the paper, and is detrimental to the reader’s understanding.  

While this might be lost because of the clarity problems described above, the model itself is also never really motivated. Why is this an interesting problem? There are many ways to create rational incentives for neurons in a neural net. Why is paying to observe activations the one chosen here? The neuroscientific motivation is not very convincing to me, considering that ultimately these neurons have to hold an auction. Is there an economic motivation? Is it just a different way to train a NN? 

Detailed Comments:
“In the of NaaA” => remove “of”?
“passing its activation to the unit as cost” => Unclear. What does this mean?
“performance decreases if we naively consider units as agents” => Performance on what?
“.. we demonstrate that the agent obeys to maximize its counterfactual return as the Nash Equilibrium“ => Perhaps, this should be rewritten as “Agents maximize their counterfactual return in equilibrium. 
“Subsequently, we present that learning counterfactual return leads the model to learning optimal topology” => Do you mean  “maximizing” instead of learning. Optimal with respect to what task?
“pure-randomly” => “randomly”
 “with adaptive algorithm” => “with an adaptive algorithm”
“the connection” => “connections”
“In game theory, the outcome maximizing overall reward is named Pareto optimality.” => This is simply incorrect. 
review_title: Requires major editing to be fit for publication. 

Meta-review: 
The reviewers have significantly different views, with one strongly negative,
one strongly positive, and one borderline negative.  However, all three
reviews seem to regard the NaaA framework as a very interesting and novel approach to training neural nets.  They also concur that the major issue with the paper is very confusing technical exposition regarding the motivation, math details, and how the idea works.  The authors indicate that they have significantly revised the manuscript to improve the exposition, but none of the reviewers have changed their scores.  One reviewer states that "technical details are still too heavy to easily follow."  My own take regarding the current section 3 is that it is still very challenging to parse and follow. Given this analysis, the committee recommends this for workshop.

Pros:
        Interesting and novel framework for training NNs
        "Adaptive DropConnect" algorithm contribution
        Good empirical results in image recognition and ViZDoom domains

Cons:
        Technical exposition is very challenging to parse and follow
        Some author rebuttals do not inspire confidence.  For example,
motivation of method due to "$100 billion market cap of Bitcoin" and in reply to unconvincing neuroscience motivation, saying "throw away the typical image of auction."
Decision: Invite to Workshop Track