Structural Credit Assignment in Neural Networks using Reinforcement Learning
Authors: Dhawal Gupta
Gabor Mihucz
Matthew Kyle Schlegel
James E. Kostas
Philip S. Thomas
Martha White
Venue: NeurIPS 2021 Poster (2021)
Keywords: Reinforcement Learning
Neural Networks
Structural Credit Assignment

Abstract: Structural credit assignment in neural networks is a long-standing problem, with a variety of alternatives to backpropagation proposed to allow for local training of nodes. One of the early strategies was to treat each node as an agent and use a reinforcement learning method called REINFORCE to update each node locally with only a global reward signal. In this work, we revisit this approach and investigate if we can leverage other reinforcement learning approaches to improve learning. We first formalize training a neural network as a finite-horizon reinforcement learning problem and discuss how this facilitates using ideas from reinforcement learning like off-policy learning. We show that the standard on-policy REINFORCE approach, even with a variety of variance reduction approaches, learns suboptimal solutions. We introduce an off-policy approach, to facilitate reasoning about the greedy action for other agents and help overcome stochasticity in other agents. We conclude by showing that these networks of agents can be more robust to correlated samples when learning online.

Structural Credit Assignment in Neural Networks using Reinforcement Learning
Dhawal Gupta, Gabor Mihuez, Matthew K. Schlegel
Department of Computing Science, Alberta Machine Intelligence Institute (Amii)
University of Alberta
[dhawal, miihucz, mkschleg Qualaerraabb....
James E. Kostas, Philip S. Thomas
Martha White
College of Information and Computer Sciences University of Massachusetts
Department of Computing Science
CIFAR AI Chair, Amii
[jekostas, pthomas @ess. umass. edu
University of Alberta
thitemmuabeert.
Abstract
Structural credit assignment in neural networks a long-standing problem, with variety of alternatives to backpropagation proposed allow for local training of nodes. One of the early strategies was to treat each node as an agent and use a reinforcement learning method called REINFORCE update each node locally with only a global reward signal. In this work, we revisit this approach and investigate if we can leverage other reinforcement learning approaches improve learning. We first formalize training neural network as -oriz-n reinforcement learning problem and discuss how this facilitates using ideas from reinforcement learning like off-policy learning. We show that the standard on-policy REINFORCE approach, even with a variety variance reduction approaches, learns suboptimal solutions. We introduce an off-policy approach, to facilitate reasoning about the greedy action for other agents and help overcome stochasticity in other agents. We conclude by showing that these networks of agents can be more robust to correlated samples when learning online.
Introduction
Training neural networks involves structural credit assignment: attributing credit (or blame) to nodes in the network for correct (or incorrect) predictions. The output from a node early in the network impacts all the outputs downstream and finally the prediction outputted the end of the network. Our goal is to adjust the weights that produced the output for this node, so that the prediction would have been more accurate. The most widely used solution for the structural credit assignment problem is backpropagation
44 namely gradient descent on the loss for the outputs.
Moving beyond backprop provides more flexibility in training neural networks. Backprop requires differentiability activations and losses for the network, as well as synchronicity for computing the gradient and updating the weights. To update node internal to the network, full feedforward and backward pass needs to be computed, with global gradient information sweeping backwards from the output. Ideally, for online agents operating real-time, with computational constraints, would have nodes that update each step. locally and asynchronously.
To make progress towards this lofty goal, we revisit old idea: treating each node as agent. Work reinforcement learning (RL), including ideas like eligibility traces, were in fact inspired by Klopf 23 and the hedonistic neuron. It is not surprising that the idea of using RL agent for each node is
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

found in early work, including the original REINFORCE algorithm [[]], which is policy gradient approach using sampled returns. Most work treating each node as an agent uses the REINFORCE update, often with baselines for variance reduction, including work on learning with spiking neurons [14] and CoAgent Networks (CoANs) 56 55 24
The work in CoANs 1) nicely formalizes the idea of a collection of agents agents-each agent corresponding to node or subset of nodes- cooperating to maximize return and 2) provides a general theorem on the validity of using the REINFORCE update. For this reason, we adopt their terminology and use CoANs to refer to networks composed of agents.
More recently, other algorithmic ideas from reinforcement learning, beyond REINFORCE, have begun to affect training of (stochastic) neural networks. The ideas of critics and baselines, which reduce the variance of policy gradient updates, have been well-developed for stochastic computation graphs [60]. This work provides unification of gradient derivations. but as yet not an investigation into practical algorithms for structural credit assignment in neural networks. Other work on learning under stochastic neurons has typically used REINFORCE heuristies to improve learning, such as straight-through estimators ( rather than improved RI. approaches. Other work on credit assignment is loosely inspired by the idea of bootstrapping in RL, including synthetic gradients 220 26 and fixed-point propagation [36].
basic method, and explored other
Overall, however, the broader space of RL algorithms has not been leveraged to learn CoANs. One reason for this omission could be that the structural credit assignment problem within the neural network has not been clearly defined as an RL problem; rather, it was simply intuitive to use REINFORCE approaches for each node. Even the theory from the original CoANs work focused on the return in the environment-sinee CoANs were used to solve a reinforcement learning problem- and did not explicitly formalize the structural credit assignment problem within the network. Another reason could be that many straightforward ideas are not effective, as we show in this work.
To facilitate the use of RL algorithms, we first formalize the structural credit assignment problem as a finite horizon RL problem. We show local policy gradient updates provide an unbiased estimate of the joint gradient for structural credit assignment, ensuring REINFORCE is sound approach. We then discuss key ideas from RL- -L_-mmely exploration and off- off-policy learning that can be leveraged to improve learning in CoANs. We show that REINFORCE can train multi multi-layered networks. but faces issues with suboptimality due to coagents learning under nondeterminism of fellow coagents. We provide an in-depth study highlighting this problem and measuring the entropy of different parts of the network. This in-depth study motivates the difficulties in using the common on-policy approaches, and we discuss and show how off off-policy learning a more promising direction. Finally, we discuss the advantages of CoANs when moving away from the standard iid learning setting, showing it can perform better than backprop on a continual learning problem with a highly correlated dataset.
1.1 Other Related Work
The literature on approaches to structural credit assignment is vast, with much of using ideas different from reinforcement learning. One category of approaches uses local updates to make activations similar to a target vector of activations, such as target propagation (8)). the method of
auxiliary variables [9] and fixed point propagation [36]. Kickback approximates the backprop update for ReLu networks, using an approximation to the gradient that allows for local updates (.). Feedback alignment [31] 38 involves using random weights, instead of the actual weights in the next layer, that avoids symmetric propagation that is thought to be biologically implausible. Weight perturbation and node perturbation approaches have been used to estimate gradients, with node perturbations emerging as the preferred approach (46). Fiete and Seung [4] showed that their node perturbation algorithm actually includes REINFORCE as special case, linking these two classes of methods. However, the connection only exists for Gaussian noise perturbations and REINFORCE; the connection is lost for other perturbations as well as for other RL algorithms.
Other work has focused on changes in weights across time Spiking neural networks and the associated meiee-tininggdeeeedeett plasticity (STDP) learning rule [32) adjusts weights based on the relative timing of activations for nearby nodes. Equilibrium propagation [4 involves a phase of propagation in the network until reaching low energy state, followed by a learning update. This work showed similarities to STDP, Contrastive Hebbian Learning (35] and Contrastive Divergence 18] Reinforcement learning updates have been used for spiking neural networks, called Reward-
modulated STDP. These updates use global reward but with local update rules (29 30), with some
interesting insights that perturbations can be beneficial to induce exploration 30 This area has
2

---

focused on delayed reward, namely assigning credit from node changes across time and across multiple updates. The local updates use node perturbation with eligibility traces to link perturbations on this step, to accuracy (rewards) at later time steps 30 34]. A more recent algorithm, called
cross propagation (58), explicitly adjusts weights back-in-time, to account for accuracy on this step, similarly to some meta-learning strategies but completely online.
There has also been some work using REINFORCE to learn activation paths through network [13 and learning when to activate parts of the network [8 []. Other work has connected structural and temporal credit assignment, but in the opposite direction from this work: specifying temporal credit
assignment as a structural credit assignment problem [
Once we use RL agents nodes, which have stochastic policies, there is clear connection to the work on stochastic neural networks. Much of this work has looked at networks with stochastic binary activations (37 42 [R], though the wider literature on stochastic computation graphs 60). Early work considered an EM-style algorithm [52) and simple alternative, called the straight-through estimator [8], that directly passes the gradient back through the node to the weights that created the activation. The straight-through estimator has recently been shown to be valid estimator for stochastic binary networks [49, and a lower-variance update has been proposed [7]
encompasses a broad range of stochastic neural networks (52] 48
Multi-agen reinforcement learning approaches tackle similar problem, in the cooperative setting- The coagents in the neural network can be seen as group of agents cooperating to produce accurate predictions, though with the notable difference that there is an ordering to the actions taken by coagents. A common approach in this area has been to use reinforcement learning agents, and consider mechanisms for coordination without centralization. Some approaches have been carefully define rewards for each agent [63 62]; ; to use a single global reward plus some noise [II]: to use independent Q-learners [51]: or to estimate global critic (39 potentially with local policy updates (5:43) When using independent Q-learners for each agent, the other agents are treated as part of the environment; consequently, the environment appears non-stationary. Hyper Q-learning [54] reduces the impact of this non- non--tationariy by estimating other agent's ' policies. These strategies do not directly extend to CoANs, but the connection to the cooperative multi-agent reinforcement learning problem could provide fruitful avenues for improved algorithms.
Finally, there have been several works empirically investigating alternative update strategies and architectures. Spiking neural networks generally have lower accuracy than standard deep neural networks, but a recent study has shown that with advances in hardware and algorithms to train spiking neural networks, this gap has become smaller [53]. Stochastic neural networks, particularly with binary activations, have been shown to be difficult train, but with some improvements to the gradient estimator, can have significant advantages, including providing a level of regularization 42). In this work, we aim provide a more comprehensive empirical investigation into the use of reinforcement learning approaches to train CoANs.
Structural Credit Assignment as a Finite Horizon RL Problem
In this section, we describe how to formalize the credit assignment problem feedforward neural network as finite-horizon RL problem. We start in the simplest setting, where we have fully connected feedforward neural network composed of l layers of size n. We assume we have inputs E X, prediction targets y E y, hidden layer activations a; = f(z;) for pre-activations 2j with activation function f. The basic idea for the finite horizon formulation is that the horizon is the number of layers, and on each step the input state for the agent is the activations from the previous step and the output is the activations for this layer or the final output prediction.
Consider the following agcnt-environment interaction. On the first step, given the sampled input X. the initial observation is Og [xc, 0] where the in the observation indicates the step-index in the finite horizon problem. The agent observes input On and then takes action ag E A that is vector of activations (or pre-activations zu) -tese actions can be discrete or continuous. The next observation is deterministically 0₃ [au. 1] (or 0 [f(ax) ), 1] = (fo)) 1)). Then the agent inputs o and outputs the activations for the next layer a and obtains next observation O2 = (a1, (or 02 [S(a,) ), 2)) This transition is Markov, because given 0₁ and a, the next observation 02 is independent of og- This interaction continues for & steps, terminating at the last layer when the final action is to output the prediction ÿ. We depict this interaction in Figure]
3

---

NN Optiiiation as Finite Horizon MDP
GV glln mid a1.., (x11)
[n] ...
0
[0--]]
[x]
(f-
Syel
Terminal
Irpurt Daia
my(ag xx)
ky(a,
Coogent Paramneterization for Continuous Nodes
Figure 1: Structural Credit Assignment within Neural Network Finite Horizon RL problem. The left figure maps components in the network to observations, actions and rewards. The right figure highlights one agent in the network and how might be parameterized.
For this interaction to be Markov, the computation of the final reward requires the input æ. The rewards received during the episode are zero, with a reward given upon termination that is a function of the error between prediction and target, such err(ū, y) (Ú - y)² with reward -err(s, y). The probability of depends on æ, and so the reward depends on x on this last step. We can obtain Markov states by simply including x in each observation. This part of the input can be ignored by the policy, as it is for most NN architectures. Formally, finite-horizon undiscounted MDP consists of state space S X {0) UX R x .1, k 1), actions 8u = [z, 0] and later states for j 2 1 are 8, [æ a,-1,j]. The transitions are deterministic, with reward of zero on each step until termination at step l giving reward rx+1 -er((. y).
IR"", where each start state
typical RL agent learns a separate policy for each horizon. This corresponds to learning one stationary policy, because the step-index is included in the state. Practically, however, these (stochas- tic) policies (a; (o)) for j E (O...., ...,, }} are updated separately, without considering the single stationary policy. If the agent is using policy gradients, then it parameterized distributions for the policies, such as Gaussians with means parameterized by the activations from the last layer. These policies can be updated using a REINFORCE update, using the gradient of the CoAN objective.
The CoAN objective corresponds to a policy gradient objective. An episode trajectory, with index information implicit and assuming actions activations, is
OD I, on: 71 = 0, 01 = [ag]..... . O2 = [a 11. a2 = G. M++1 = crr( v), Termination.
Because (ao; 01, . ak (r) N (as x)) I my (ajlay-1). the probability of this trajectory is p(x)) ((rr)(aa)) (ag. ai,..., . ( (()l()))
This is because p(z) gives the probability of the start state; (ag. ai..... 4x/2 ax/²) gives the probability of the trajectory from 8o because the state outcomes are deterministic given the action (activation or pre-activation); and p(yla) 2)) defines the probability of the reward on termination, since the target is stochastic. This provides straightforward policy
gradient objective, for undiscounted return def err((a,,)
max / p(=)) a......, (x(x))((la=)) G dr das da dy max IF (ze.)) [G]
(1)
If we assume each policy nj has parameters 0,. then the stochastic gradient of this objective separates out into the following stochastic gradients for each policy separately: GV o, In æT ma (a,lo,). We show this formally Proposition D in Appendix A A similar result has been shown for CoANs used in the RL setting [24] Theorem 1] and for stochastic computation graphs [60] Theorem 2]; we include the result specifically for this case because it avoids much of the complications from those other works.
The locality of the policy gradient means policies can be updated locally with their own gradients, with shared global return signal. We can also easily incorporate control variates reduce the variance of the gradient, called baselines, The update is (G V )), In ITy (a,lo), where the learned baseline V(o,) estimates expected return given oy: G - V(o,) corresponds to the advantage for the actions selected. Several different critics have been proposed for stochastic computation graphs [60]; we discuss how to make similar baselines for this finite horizon problem in Appendix B
The CoAN is trained as stochastic network, but is evaluated under the standard (deterministic) backprop setting. The question we is: can a CoAN, trained with local updates, produce predictions with similar accuracy to those produced by an NN, trained with backprop? We therefore test the

---

CoAN using the greedy actions from the co-agents, to provide deterministic predictions. Effectively. we are comparing the parameters 0 produced by the CoAN those produced by backprop
We described the formalism for the simpler feedforward setting, to facilitate understanding. All the ideas, however, extend to more generic acyclic networks, because every acyclic network has a topological ordering on nodes. The state input for each coagent still consists of the input nodes, At each time step, whichever coagents have all their nodes evaluated-nambe have their state input available-can produce their action. This propagates forward until prediction is produced.
Remark:
3 Issues with Stochasticity in On-policy Updates for CoANs
In this section, we show that though CoANs with REINFORCE can learn, they are significantly hindered by stochasticity in other coagents. Variance is a well-known problem in stochastic networks. Here, we highlight that even for a variety of variance reduction approaches, learning plateaus at a suboptimal point. The issue is less severe with discrete actions actions-binary binary rather than continuous activations-bul but a significant gap remains when contrasting to idcalized (low-variance) gradients.
3.1 Experimental Details
We investigate CoANs on problems where backprop is known to perform well, to provide a strong baseline and facilitate understanding the behavior, and potential issues, when learning in CoANs. We expect backprop to outperform CoANs here, and ask: how much worse are CoANs compared to backprop, and can we close the gap with simple variance reduction techniques? To investigate this question we use two well-studied datasets: MNIST 27 for classifying handwritten digits, and the Boston Housing Dataset from UCI.
Where possible, we matched the architecture and optimization choices for backprop and the CoAN learner. We test both strategies using single and louble-layer neural network, with 64 hidden nodes and ReL.U activations. Each node in the CoAN is a single coagent using a Gaussian distribution with parameterized mean and a fixed standard deviation, set system-wide through a systematic sweep over E 10.1, 0.5, 1.0, 2.0, 4.0, 8.0, 16.0]. The feedforward procedure samples actions from these policies and then applies the layer's activation function (i.e. ReLU for hidden layers or the identity/softmax function for the output layer). Both use RMSProp [57]. with fixed 8 = 0.99 and stepsizes swept for E {2- -22 ...,2 2-153. We use mini- bini-batch gradient descent with batch size 32 for MNIST with 50 epochs, and full gradient descent for the Boston Housing Dataset with 10k epochs. Hyperparameters are chosen from performance on a validation set held out from the training set: 10K for MNIST and 51 samples for Boston Housing. Results are averaged over 10 independent runs and compared using the area under curve (AUC).
Along with the standard CoAN with no baseline, we also test the effectiveness of three baselines. The Global baseline (Co-G) maintains a scalar running average of the global loss function parameterized by the rate of decay (fixed at 0.99). The State Global baseline (Co-SG) learns a parameterized value function associated with each input/ state to the complete network and tries to predict the loss. The State Layer baseline (Co-SL) is per layer baseline, where we learn a parameterized value function for each layer based on its input learned using Monte Carlo updates from the global loss. More details on baselines are in Appendix B and pseudocode in Appendix H
3.2 Results
We report the performance of the best performing parameters on a held-out test set with the same size as the validation set in F (a). At glance, it clear that backprop outperformed all the CoANs. It is surprising that there is no improvement using the local baselines as compared to the global baseline. natural question from these results is if the gap between CoANs and backprop due
'The optimal parameters for the (surrogate) policy gradient objective above may not be the same as those for the deterministic network. Namely, the co-agents minimize E((Ý- - y)²] across pairs (æ,, y). where V is stochastic due to the stochastic co-agent policies. This is in contrast to minimizing the deterministic output (E[Ý)- - y)² or the loss where each co-agent takes a greedy action. The co-agents may actually adjust their outputs. to mitigate risk from the stochasticity in fellow coaatess. Somewhat surprisingly, we find in our experiments that there is no such gap, and optimizing this surrogate produce optimal solutions.
5

---

Laper
(P)
Lager
Siginiid
MNST
BaLU
Epectra
Specta
BOSTON
Co
Co-50
Spocka
Spacts
Figure 2:
All curves are averaged for 10 runs with standard error bars. (a) Training different variants of CoAN's and backprop on MNIST and Boston Housing with one and two layer NNs. (b) Experiments highlighting issues with nondeterminism in coagents. The upper graph depicts performance with different coagent partitionings of the network. The lower graph depicts the training curve for layer coagent network with C1-C1-CI partitioning for different amounts of averaging i.c., [1,5, 10, 25, 50]. (c) Failures of different strategies gradually reduce nondeterminism of coagents overtime, on Boston Housing.
to the optimization process, or poor stationary point of the CoAN itself. Experiments running the CoAN for longer showed only very slow improvement. Before this experiment, we hypothesized that high variance updates would negatively impact the CoAN optimization, but the variance reduction schemes used above did not improve performance at all.
To investigate the role of stochasticity in coagents, test different partitioning schemes of the two-layer coagent network. We can treat the whole network as one coagent, and use backprop within that agent (called C1). We final layer and one that learns the weights to produce the prediction (called C2-C1). CI.. We have four partitioning schemes, where we always have prediction coagent to keep the objective consistent, labeled: CI-CI-CI, C2-C1, C1-C2, and C3. C1-C1-C1 has a coagent for each layer, and C1-C2 uses a linear coagent for the first layer and single layer neural net coagent for the next.
two coagents: that outputs the activations for the
In Figure 2 (b, upper), we see the deterministic network C3 performs similarly to the backprop network. As add stochasticity back to intermediary layers, performance degrades significantly. This highlights that the stochasticity in the last layer is not a culprit, and that variance lower for C1-C2 where stochasticity is in the first layer than for C2-C1 where stochasticity is in a later layer. In either case, the introduction of stochasticity in intermediate layers starkly reduces performance.
To ascertain that the issue is due to variance, rather than poor stationary points, we can use an idealized gradient that is not practical for our desired learning setting, but can act as a control for this experiment. We can obtain low-variance gradient estimate simply by sampling the stochastic network many times, for the given pair (x, y). Such an update non-local-rrquiin coordination-ant so is not a practical approach to reduce variance for our setting. But this averaged gradient, particularly with increasing samples for the average, gives insight into the optimization surface for these networks as well as the magnitude of the variance. Figure (b, lower) shows the effect on learning with increasing samples used in the average gradient, for the two layer network. With increasing samples, the training of the CoAN significantly improves, nearly matching the learning speed of backprop with 50 samples. This highlights that the surrogate objective used by the CoAN results in reasonable solutions when testing the greedy actions given by the CoAN. It shows that there is significant variance in the gradient, and that baselines are not reducing that variance.
significant
3.3 Failures of Simple On-policy Approaches to Reduce Nondeterminism
natural next step to consider simple strategies to gradually reduce the stochasticity of coagents. For example, it is common in RI. to have a decay schedule for the exploration parameter. Similarly here, instead of using a fixed variance parameter for coagents, we can gradually decay this parameter and so allow coagents to gradually learn good actions under nearly greedy actions from other coagents.

---

Layer
Layer
Accuracy
53%
Co-O
Tran
Spardhy
ST
Loger
Epooto
Epecto
Layer
Epoctt
(b)
Experiments on MNIST. using 64 nodes per layer. (a) Learning curves for discrete agents (ST, Co, Co-G) with single and double-layers, averaged for 10 runs, for 50 epochs. (b) Learning progress of three layer network for 10K epochs. with entropy (solid line) and sparsity (dotted line) of each layer, along with accuracy (scaled to [0, I), which matches backprop (with accuracy > 99%)..
Figure 3:
We tested two strategies to enable the variance parameter in the Gaussian distributed actions to decrease with time. The first strategy is to learn a a per node, using bias unit per node, passed through either a sigmoid, ReLU exponential obtain a non-negative number. The second is to use a decay rate for the a of the whole network. In both cases we initialized a,, with the best value found in the original experiments (i.e. a 4.0). Unfortunately, these natural strategies to remove stochasticity in coagents do not improve performance, as can be seen in Figure (c)). One possible reason is that for smaller a of 0.9 and 0.99 the gradient Vo ((alo) can get very big for low probability sampled actions. This instability can be avoided with discrete actions, or with off- off-policy learning, both of which we investigate next.
3.4 Discrete Actions Helps Control Stochasticity, But Not Enough
Here we study discrete coagents with Bernoulli discrete actions, and their stochasticity over time. Discrete networks provide an easier way to handle stochasticity in their policy as a softmax parame- terization adapts stochasticity when learning, and can become fully deterministic. As a baseline, we do gradient backpropagation via straight-through (ST) estimators [ 50), because standard backprop cannot be used for discrete nodes. In this set of experiments, we again use REINFORCE coagents, and test on MNIST. We measure the entropy of the coagents over time, well as the sparsity of the representation to ensure reasonable levels of activation.
In Figure 3 (a) we can see that CoANs actually perform better than the baseline, the ST estimator, and here the baseline had bigger positive effect (Co-G versus Co). ST estimators have difficulties when learning in more than single layer (8) probably due to misaligmment [50]. For Co-G, the ability to better control the entropy seems to help: in Figure the drops in entropy-earlis for layer one and then at 200 epochs for layer two and three-cause a sudden rise in accuracy. However, the entropy does not fully decrease and there some unintuitive behavior. The first layer becomes deterministic faster, which is surprising it relies stochastic actions of downstream agents. The entropy for layer two and layer three also decreases, but eventually, the entropy starts to creep back up while maintaining or slightly improving the accuracy. We note that these networks are able to match the performance of a standard NN with backprop (i.e., accuracy > 99%) but it takes a very large number of training steps (approximately 10K epochs).
We provide more in-depth investigation into the stochasticity under discrete actions, including using more discrete actions per coagent and training with (randomized) subsets of coagents fixed, in Appendix C Overall, we find that the former significantly increases performance in llayyer networks for both REINFORCE coagents and especially action-value methods; however, the latter approach does not substantially reduce the entropy, nor improve performance.
4 Off-Policy Learning to Learn Nearly Deterministic CoAgents
The chief difference between using RL and typical optimization approaches, both for SCGs and standard NNs, is that we can learn off-policy. When training a neural network, it rarely the case that the accuracy of predictions matters when doing update. Rather, this setting matches the fully offline learning setting-thh the pure exploration agent needs to maximize reward while learning. This highlights that can also many different
of the online setting where the
7

---

exploration approaches, to gather useful data about how to adjust the policies for more accurate predictions. The behavior could choose to make a poor prediction, to gather experience that is more useful for improving accuracy than if the on-policy best prediction was used. This separation is in stark contrast to methods like backprop.
In this section, we show how to learn critics off-policy, and that this improves on using on-policy crities. We start by describing the algorithm, and then provide results in MNIST for the case of discrete coagents. We provide additional results in Appendix [ showing that on-policy action-value methods do not improve on REINFORCE, further motivating the shift to off- off-policy learning.
4.1 An Off-Policy Algorithm for CoANs
The first step to modify the REINFORCE update, to use an action-valus critic. Instead of using sampled return G, we can estimate the expected return Q, aj), for the coagent taking action a, given input oj. namely the previous hidden layer aj-1- The update then uses (Q,(o,, aj) (2)),)) Ve, In rT (a,lo)), where for the final layer we do not learn a critic and simply using the immediate reward (i.e., the negative of the error). These action-values can be updated on on-policy, each time the network is queried, using Sarsa update
((0) ((1), + a(0 + Q az+1) Qソ(oj, a3) where Qj (g) can simply be linear function of o,. a, namely linear function of (a,-1. or could itself be a small neural network.
VQ, aq)
This strategy, however, introduces bias for two reasons. First, estimating action-values means we have some error in our expected return estimate, due both to estimate error and approximation. Second, the local action-values are actually tracking a non-stationary target. They estimate the expected return for an action, where the expectation is taken over the input and output as well as the actions of the other coagents. Further, the coagent is attempting to learn how to select actions, given stochastic action selection by the other coagents rather than the best action (greedy action) for each co-agent. This nonstationarity and difficulties in credit assignment is well-recognized as an issue in multi- -agent reinforcement learning [54] 15 43 However, in our setting the known structure between agents means we can more easily obtain solution, than an unstructured collection of cooperating agents.
The key is reason about greedy actions of downstream coagents, rather than the action they actually took. The update has a small modification, to instead use a maximum over values in the next layer (( 0(9); ; + a(0 + max Qj+1(o5+1, a') Q,(0,, (,)) ag)
Given the input, the agent asks: what is the value of each action, given the maximal actions will be taken for downstream layers? This update bootstraps only on the action-value in the next layer, but the update for that Qj+1 1 also bootstraps off of the max in the next layer. Therefore, each action- value starting from the end of the network is learning about maximal action-values for downstream coagents, and propagating that information backwards. This approach directly exploits the known Markov structure of the credit assignment problem, and so should learn more efficiently than using structureless algorithms like REINFORCE.
The coagents do not need to reason about the greedy actions for upstream coagents, because action- values are conditioned on inputs produced by those coagents. For a given activation from the previous layer a,-1, the coagent learns π (a, ay-1). Under sufficiently high capacity policy parameterizations, the coagent can simply learn what to output for a variety of different inputs, including those that are more optimal for the given input æ. straightforward to show that in Equation ( under unrestricted policy parameterizations, each policy can be optimized assuming these greedy action- values for downstream layers.
Practically, however, our coagent policies are likely to be under-paramett parametrrraeteizeedd. If an upstream coagent provides wide range of activations a then the policy has to trade-off function approximation across this large input space. Ideally. it would focus function approximation on the aj-1 that are likely to produce good predictions ÿ. This trade-off a common issue in policy gradient methods. We can incorporate a reweighting, to increase the importance of an update for an aj-1 that is more probable-more greedy from the upstream coagent, and decrease the importance of an update for lower probability 11-1- Such importance is already implicit, because aj-1 1 are sampled proportionally to the probabilities. For this reason, we do not address further here, but note that there are some insights for state reweightings in policy gradient methods to improve solutions 12]19]
8

---

Layer
d=0.95
Cb-60 Policy
4:2.00
Co-AC
Co/6
Epocha
Figure 4: (a) Learning curves comparing on policy and off policy actor critic. Averaged for 10 runs, for 3 layer neural network with 64 nodes per layer. (b) CL experiments with problem difficulties at = 0.98 dushed lines and solid lines for d = 1.0 comparing backprop and CoAN's. (c) Sensitivity plots in CL problem for learning rate (a), for problem difficulty of d = 0.95.
4.2 Results with Off-Policy CoANs
We perform a set of experiments where we estimate Qo(o,, (,00,aa] az) function for each layer in the CoAN, using the on-policy and off-policy approaches described in Section 4.1 We allowed our agents to select from different forms of critics which include linear and single-layered neural nets with (64, 128 & 1024) nodes. We found that NN critics worked best with 1024 nodes and learning rate twice the value of the CoAN. We keep the other hyperparameters the same from Section 3.4 Note that here our goal is to understand if crities practically, depending on the setting, there may be stronger limitations on the critics.
help, and allow for these larger crities per coagent;
Figure 4 (a) presents the results for on on-policy (Co-AC) and off off-policy (Co-AC Off Policy) critics. We find that off-policy critics help improve performance, particularly later in learning, whereas the on-policy critic is plateauing at lower point. As note, the performance with critics is actually worse than that REINFORCE, likely due to bias the crities. Better approaches to learning the critic off-policy, including using methods like replay, should help close this gap. Our goal in this experiment was primarily to contrast the on on-policy and off off-policy critics, to highlight that off- policy learning is promising direction towards addressing the nondeterminism issue in CoANs.
5 Continual Learning Experiments
In the previous sections, we chose settings where backprop is effective, to make the results more interpretable and start from the standard learning setting. The motivation for CoANs, however, goes much beyond the standard setting. The goal is to facilitate learning in other settings, particularly in continual learning settings with correlated data and the need for real-time learning and decision- making. CoANs also naturally facilitate asynchronous inputs and updating [24], as well as learning with recurrence. Traditionally stochastic methods have known regularize and help avoid local minima; we provide a small experiment showing this advantage of CoANs in Appendix D
In this section, we investigate the utility of CoANs for prediction when learning online with highly temporally correlated inputs. We expect CoANs to be less prone to failure than backprop. which can completely overwrite previous learning when learning on correlated data. Because coagents are RL agents, the policies should better track and adapt to changes in the environment.
To measure coagents' ability learn online with various degrees of temporal correlation, we examine the performance of Co-G on the Piece Random Walk Problem introduced in [41]. with the same parameter settings and target function. In this dataset, the correlation difficulty parameter d [0,1] controls the amount of temporal correlation within the data: O stands for iid data points, while indicates fully temporally correlated dataset.
We first train Co-G and backprop for 180,000 training steps, with d E (0, 0.85, 0.95, 1). and test every th step on
set of 1,800 iid samples, with the best hyperparameters picked on the 2-
validation set of equal size. Standard deviation for Co-G is swept ø E 22 222.220 2 22 23 2t). For both algorithms, stepsize values are swept a {2- 14 2 13 2 122111166
2-2-7, 2.27- the batch size is fixed at 32, number of units at 50, and number of hidden layers at 1. Co-G is trained using the RMSProp optimizer, while backprop is trained using the Adam optimizer, both with standard 33 values. Then, using the chosen hyperparameters, we allow the agent to continue learning up to 320,000 training steps on E (0.98, 1). to gauge the long run performance on extremely correlated data. Results are averaged over 200 runs.

---

Backprop outperforms Co-G on iid samples, and achieves lower error on d=0.85. Co-G begins to display an edge at d=0.95, where it achieves similar performance to backprop, and at d=1, Co-G performs better and learns steadily, while backprop is incapable of learning. In addition, the sensitivity plot for this experiment in Figure depicts that while backprop's 's performance is largely dependent on the correct alpha parameter, Co-G's performance remains approximately the same across the swept alpha values. Co-G is also similarly insensitive to its a hyperparameter, as shown in the Appendix B
Looking at long run performance after 180,000 steps, we see two interesting phenomena. Co-G is able to continue to steadily learn on the highly correlated dataset, unlike backprop. But, for d=0.98, backprop actually starts to match the performance of Co-G. These results, however, are for carefully swept hyperparameters. and we see that backprop is quite sensitive to the stepsize. Under this heavy correlation, therefore, we find that CoANs provide more robust performance, in that they provide steady progress from the very beginning of learning and are much less sensitive to the stepsize.
6 Limitations and Next Steps
Once we have formalized this problem as a finite-horizon RL problem, it facilitates applying variety of RL algorithms. The approaches in this work are arguably the simplest first choices, and do not incorporate useful advances in exploration strategies, policy gradient methods, off- off-policy learning and replay approaches. Our CoANs relied solely on the stochasticity in the coagents to explore. Such randomized exploration strategies are unlikely to be as efficient directed exploration approaches, even when restricted to only those that learn values and not models, such as those using upper confidence bounds values 16 40 25 and information-dirrccing sampling (45) Such directed
exploration strategies have also recently been developed for policy gradient methods (2)
There have also been many recent advances to policy gradient methods, with better theoretical understanding and improvements for the off-policy setting. Of particular relevance work that examines the importance of state weightings and dealing with distribution shift 13, which is key for the off off-policy setting [9]. Another important insight leveraging connections between approximate policy iteration and policy gradient methods, to obtain performance guarantees [12 59 [0]. This perspective assumes that an explicit policy and explicit values are learned, and facilitates off-policy learning using the learned values. One of these methods, called SBEED [2]. further exploits recent advances in gradient- based methods for learning values off-policy.
All of these more advanced approaches rely on value estimation. Value estimates help direct exploration, by facilitating reasoning about uncertainty and incorporating optimism. They naturally facilitate off-policy learning and replay, because they allow us to learn from short experience tuples. They allow each agent to reason about counterfactual outcomes, and perform more policy updates in the background, asynchronously. The bias, however, currently precludes obtaining these benefits; we as yet do not know why it is so harmful. When using Actor-Crite in other settings, the bias in the critic does not seem to be so detrimental. One issue here may be that PG methods have not been specially designed for finite-horizon problems, where a different policy is used each step. For us, the changing policies make it less straightforward use standard reductionn strategies estimate critics, like eligibility traces. Understanding the influence of bias in the critic for structural credit assignment, and how overcome it, is one of the most important open questions for CoANs.
7 Conclusion
In this paper, we investigated the use of reinforcement learning (RL) for the structural credit assign- ment problem in neural networks. We formalized this problem finite-horizon MDP, and showed that local policy gradient updates for each node (coagent) provide an unbiased estimate of the joint policy gradient for all nodes. We show that the basic local policy gradient update for this coagent network (CoAN) can learn- even under difficult learning settings like highly correlated data- but that it plateaus at suboptimal solutions. Through a set of targeted experiments, we highlight that the stochasticity amongst the coagents results in this suboptimality, and that it cannot be mitigated with standard variance reduction strategies or attempts to gradually reduce stochasticity of the coagents towards deterministic policies. We highlight that off-policy learning can naturally be applied to this problem, through the use of off-policy critics. We show that this strategy promising, but that much more work needs to be done to improve the learned critics and mitigate bias.
10

---

Acknowledgments and Disclosure of Funding
This work was supported by NSERC Discovery, IVADO, CIFAR through CCAI Chair funding and by the Alberta Machine Intelligence Institute (Amii). This research was funded in part by NSF award #2018372. The authors would also like to thank Kirby Banman.
References
[1] Yasin Abbasi-Yadkorf Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellert Weisz. POLITEX: Regret Bounds for Policy Iteration using Expert Prediction. In International Conference on Machine Learning, 2019.
[2] Alckh Agarwal, Mikael Henaff, S. Kakade, and Wen Sun. PC-PG: Policy Cover Directed Ex- ploration for Provable Policy Gradient Learning. In Advances in Neural Information Processing Systems, 2020.
[3] Alekh Agarwal, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. On the Theory of Policy Gradient Methods: Optimality, Approximation, and Distribution Shift. 1rXiv:1908.00200 2020.
[4] Adrian K. Agogino and Kagan Tumer. Unifying temporal and structural credit assignment problems. In Proceedings of the International Joint Conference on Autonomous Agents and Multi-Agent Systems, 2004.
[5] David Balduzzi, Hastagiri Vanchinathan, and Joachim Buhmann. Kickback cuts backprop's red-tape: Biologically plausible credit assignment in neural networks. Proceedings of the AAAI Conference Artificial Intelligence, 2015.
[6] Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computa- tion in neural networks for faster models. arXiv:1511.06297, 2015.
[7] Yoshua Bengio. How Auto-Encoder Could Provide Credit Assignment in Deep Networks via Target Propagation. 1407 2014.
[8] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation. arXiv:1108.33333 2013.
[9] Miguel Carreira-Perpinan and Weiran Wang. Distributed optimization of deeply nested systems. In Artificial Intelligence and Statistics, 2014.
[10] Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, A. Rupam Mahmood, and Martha White. Greedification Operators for Policy Optimization: Investigating Forward and Reverse KI. Divergences. arXiv: i0l::02077.228 2021.
[11] Yu-Han Chang, Tracey Ho, and Leslie Pack Kaelbling, All learning is local: Multi-agent learning in global reward games. Advances in Neural Information Processing Systems, 2003.
[12] Bo Dai, Albert Shaw, Lihong Li, Lin Xiao, Niao He, Zhen Liu, Jianshu Chen, and Le Song. SBEED: Convergent Reinforcement Learning with Nonlinear Function Approximation. In International Conference on Machine Learning. 2018.
[13] Ludovic Denoyer and Patrick Gallinari. Deep Sequential Neural Network. arXiv:1410.0510 2014.
[14] Ila R. Fiete and H. Sebastian Seung. Gradient Learning in Spiking Neural Networks by Dynamic Perturbation of Conductances. Physical Review Letters, 2006.
[15] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[16] Robert Grande, Thomas Walsh, and Jonathan How. Sample Efficient Reinforcement Learning with Gaussian Processes. In International Conference on Machine Learning, 2014,
11

---

[17] Shixiang Gu, Sergey Levine, Ilya Sutskever, and Andriy Mnih. Muprop: Unbiased backpropa- gation for stochastic neural networks. In International Conference on Learning Representations. 2016.
[18) Geoffrey E. Hinton. Training products of experts by minimizing contrastive divergence. Neural Computation, 2002.
[19] Ehsan Imani, Eric Graves, and Martha White. An Off-policy Policy Gradient Theorem Using Emphatic Weightings. In Advances in Neural Information Processing Systems, 2018.
[20] Max Jaderberg, Wojciech Marian Czarnecki, Simon Osindero, Oriol Vinyals, Alex Graves, David Silver, and Koray Kavukcuoglu. Decoupled neural interfaces using synthetic gradients. In International Conference on Machine Learning, 2017.
[21] Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In International Conference on Machine Learning. 2002.
[22] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations, 2015.
[23] A. Harry Klopf. The Hedonistic Neuron: A Theory of Memory, Learning, and Intelligence. Hemisphere Publishing Corporation, 1982.
[24] James Kostas, Chris Nota, and Philip Thomas. Asynchronous coagent networks. In International Conference on Machine Learning, 2020.
[25] Raksha Kumaraswamy, Matthew Schlegel, Adam White, and Martha White. Context-dependent upper-confidence bounds for directed exploration. Advances in Neural Information Processing Systems, 2018.
[26] Benjamin James Lansdell, Prashanth Ravi Prakash, and Konrad Paul Körding. Learning to solve the credit assignment problem. In International Conference on Learning Representations, 2020.
[27] Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010.
[28) Dong-Hyun Lee, Saizheng Zhang, Asja Fischer, and Yoshua Bengio. Difference target propa- gation. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD, 2015.
[29] Robert Legenstein, Dejan Pecevski, and Wolfgang Maass. Learning Theory for Reward- Modulated Spike-Timing- Tennnn-Deeee-ne Plasticity with Application Biofeedback. PLOS Compu- tational Biology, 2008
[30] Robert Legenstein, Steven M. Chase, Andrew B. Schwartz, and Wolfgang Maass, A Reward- Modulated Hebbian Learning Rule Can Explain Experimentally Observed Network Reorganiza- tion in Brain Control Task. Journal of Neuroscience, 2010.
[31] Timothy Lillicrap, Daniel Cownden, Douglas B. Tweed, and Colin Akerman. Random synaptic feedback weights support error backpropagation for deep learning. Nature Communi- cations, 2016.
[32] Henry Markram, Wulfram Gerstner, and Per Jesper Sjöström. Spike-Timing ity: A Comprehensive Overview. Frontiers in Synaptic Neuroscience, 2012.
Baake-Timing-Dependeee Dependent Plastic-
[33] Thomas Merkh and Guido Montúfar. Stochastic Feedforward Neural Networks: Universal Approximation. arXiv:1910.09763, 2019.
[34] Thomas Miconi. Biologically plausible learning in recurrent neural networks reproduces neural dynamics observed during cognitive tasks. elifee, 2017.
[35] Javier Movellan. Contrastive Hebbian Learning in the Continuous Hopfield Model. Connec- tionist Models, 1991.
12

---

[36] Somjit Nath, Vincent Liu, Alan Chan, Xin Li, Adam White, and Martha White. Training recurrent neural networks online by learning explicit state variables. International Conference on Learning Representations, 2020.
[37] Radford M. Neal. Learning stochastic feedforward networks. Technical report, 1990.
[38] Arild Nøkland. Direct feedback aligmment provides learning in deep neural networks. In Advances in Neural Information Processing Systems, 2016.
[39] Frans A. Oliehoek, Matthijs T. J. Spaan, and Nikos Vlassis. Optimal and approximate Q-value functions for decentralized POMDPs. Journal of Artificial Intelligence Research, 2008
[40] lan Osband, Benjamin Van Roy, Daniel Russo, and Zheng Wen. Deep Exploration via Random- ized Value Functions. Journal of Machine Learning Research, 2019.
[41] Yangchen Pan, Kirby Banman, and Martha White. Fuzzy tiling activations: A simple approach learning sparse representations online. In International Conference on Learning Representations, 2021.
[42] Tapani Raiko, Mathias Berglund, Guillaume Alain, and Laurent Dinh. Techniques for learning binary stochastic feedforward neural networks. In International Conference on Learning Representations, 2015.
[43] Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Foerster, and Shimon Whiteson. QMIX: monotonic value function factorisation for deep multi- agent reinforcement learning. In Proceedings of the International Conference Machine Learning, 2018.
[44] David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. Learning representations by back- procaaappgggning errors. Nature, 1986.
[45] Daniel Russo and Benjamin Van Roy. Learning to Optimize via Information-Directed Sampling. In Advances in Neural Information Processing Systems, 2014.
[46] Hiroshi Saito, Kentaro Katahira, Kazuo Okanoya, and Masato Okada. Statistical mechanics of structural and temporal credit assignment effects on learning in neural networks. Physical Review. E, Statistical, Nonlinear, and Soft Matter Physics, 2011.
[47] Benjamin Scellier and Yoshua Bengio. Equilibrium propagation: Bridging the gap between energy-based models and backpropagation. Frontiers in Computational Neuroscience, 2017.
[48] John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient estimation using stochastic computation graphs. In Advances in Neural Information Processing Systems, 2015.
[49] Alexander Shekhovtsov and Viktor Yanush. Reintroducing Etraigh-TThroug Estimators as Principled Methods for Stochastic Binary Networks. arXiv:2006.0688 06880, 2021.
[50] Alexander Shekhovtsov, Viktor Yanush, and Boris Flach. Path sample-analytic gradient estima- tors for stochastic binary networks. In Advances in Neural Information Processing Systems, 2020.
[51] Ming Tan. Multi-Agent Reinforcement Learning: Independent vs. Cooperative Agents. In International Conference on Machine Learning, 1993.
[52] Yichuan Tang and Ruslan Salakhutdinov. Learning stochastic feedforward neural networks. In Advances in Neural Information Processing Systems, 2013.
[53] Amirhossein Tavanaci. Masoud Ghodrati, Saeed Reza Kheradpisheh, Timothee Masquelier, and Anthony S. Maida. Deep Learning in Spiking Neural Networks. Neural Networks, 2019.
[54] Gerald Tesauro. Extending Q-Learning to General Adaptive Multi-Agent Systems. In Advances in Neural Information Processing Systems, 2003.
13

---

[55] Philip S Thomas. Policy Gradient Coagent Networks. In Advances in Neural Information Processing Systems, 2011.
[56] Philip S. Thomas and Andrew G. Barto. Conjugate markov decision processes. Proceedings of the International Conference Machine Learning, 2011.
[57] T. Tieleman and G. Hinton. Lecture RmsProp Divide the gradient by running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
[58) Vivek Veeriah, Shangtong Zhang, and Richard S. Sutton. Crossprop: Learning representations by stochastic meta-gradient descent in neural networks. In Machine Learning and Knowledge Discovery in Databases European Conference, ECML PKDD, 2017.
[59] Nino Vieillard, Tadashi Kozuno, Bruno Scherrer, Olivier Pietquin, Remi Munos, and Matthieu Geist. Leverage the Average: an Analysis of KI. Regularization Reinforcement Learning. In Advances in Neural Information Processing Systems, 2020.
[60] Théophane Weber, Nicolas Heess, Lars Buesing. and David Silver. Credit Assignment Tech- niques Stochastic Computation Graphs. The International Conference on Artificial Intelligence and Statisties, 2019.
[61] Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforce- ment learning. Machine Learming, 1992.
[62] David H Wolpert and Moffett Field. Optimal Payoff Functions for Members of Collectives. Modeling complexity economic and social systems, 2002.
[63] David Wolpert, Kevin R. Wheeler, and Kagan Tumer. General principles of learning-basee multi-agent systems. In Proceedings of the Annual Conference on Autonomous Agents, 1999.
14


Reviews:
[{"review_id": "yllDRheKoOL", "reviewer": "Reviewer_wqi1", "summary": "This paper presents a study that investigate RL methods for the credit assignment problem in neural network. In this study, the credit assignment problem is formalized as a finite horizon RL problem. Subsequently, it is empirically demonstrated that stochasticity of coagents’ policies hinders the learning performance when REINFORCE is used to train coagents. To address this issue, a Q-learning like off-policy algorithm is proposed. The experimental results show that the proposed off-policy algorithm outperformed on-policy algorithm. In addition, the proposed off-policy algorithm outperformed backpropagation in a continual learning task.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strong points:\n-\tThe empirical results on the effect of stochasticity are interesting.\n-\tThe proposed algorithm outperformed backpropagation in the continual learning task.\n\nWeak points:\n-\tThe paper is not well-organized.\n-\tSome more experimental results are necessary to support the claims.\n\nI think that the empirical results on the stochasticity of coagents are interesting, but the paper is not well-organized and hard to follow in some parts. I think that the paper requires major revision to improve the clarity. I recommend to separate the algorithmic discussion and the experimental section, instead of writing finding in a chronological order.\nIn addition, it is necessary to add some more experiments to support claims.  I listed points to be improved in the following.\n\nDetailed comments\n-\tKostas et al. [2020] also formalized the credit assignment problem as RL, and I do not see clear difference between the formulations in this paper and their work. \n-\tPresentation in this paper is not well-organized, and it is hard to follow in some parts. For example, I’m not sure whether the action is assumed to be discrete or not in Section 4. If the action is discrete, it is natural to apply the Q-learning algorithm without an explicit model of a policy. Please clarify whether the action is discrete or not and discuss the relation to the Q-learning algorithm and “Co-AC off-policy”, which is proposed in Section 4.1.\n-\tIn Figure 2, I guess MC represents on-policy Monte Carlo, but there is no description that explains what MC represents.\n-\tIn equations below the line 304 and 317, I do not understand we have\n … \\alpha( 0 + max Q … instead of \\alpha ( r + max Q…\n-\tIn the continual learning experiment, the proposed method is referred to as “Co-G”. I expected that “Co-AC off-policy” should be evaluated in this experiment. In addition, variants of RL methods, including “Co-AC on-policy” and “on-policy Monte Carlo”,  should also be evaluated in the continual learning task.\n\n=== comments after author response ===\nAs my questions were answered in the author response and discussions, I raised the score. Although I'm leaning to acceptance after discussions, I would like to remind the author that the paper requires major revision to improve the clarity.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Although this study presents a method for credit assignment in neural networks, the results do not indicate that the backpropagation will be replaced with the CoAN in near future. I recommend to clearly describe this point for audience who are not experts in this field.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "sEx6FzR9n1", "reviewer": "Reviewer_JMW8", "summary": "This paper studies credit assignment problem in neural networks with RL algorithms. They first formulate the problem as a finite-horizon MDP so that they can apply RL. They find nondeterminism in the problem causes suboptimal solutions and propose to use off-policy RL with a critic to resolve it. ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper's main contribution is to understand off-policy RL is better than the previous policy gradient method (REINFORCE) in credit assignment problems. The authors did a thorough analysis and explained well about the previous difficulty and why they introduce critic into this problem. In the view of an RL researcher, the algorithmic contribution is quite limited. Hence, the significance of this paper is not high enough for the bar of this conference.\n\nBut from the angle of credit assignment, these might be crucial findings and opens a welcoming avenue for future research. My judgment might not be correct due to the fact I only know RL well. \n\nAs the authors mentioned, there might be multiple aspects from RL that can be applied to this problem as well. My suggestion would be to add those to the current framework as well and provide somewhat more convincing results would make this paper stronger.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "It seems that the authors only test on very simple neural nets. My question would be: is this the tradition in this area or it is a proof of concept? If the latter, I believe more experiments with modern architecture is encouraged. \n\nAll the figures are not self contained. For example, some of the legends are missing. The difference between dashed lines and solid lines are not clearly readable within the figures.\n\nSetting, experimental results and analysis are intertwined together without clear separation.  \n\nThe results are not showing the potential for utilizing the proposed technique in other areas. \n\n\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "jvkquioTeKF", "reviewer": "Reviewer_To4V", "summary": "The paper investigates using reinforcement learning to train neural networks. Specifically, it considers formulates the problem in which individual layers are treated as steps in a finite horizon partially observable MDP, and explores using either a global agent or individual co-operating agents at each layer to maximize the reward (the negative final error).\n\nThe paper then presents an empirical study of using REINFORCE in this cooperative agent framework to train one or two hidden layer networks on MNIST/Boston Housing while varying different parameters like the policy variance, different partitioning schemes of the network into agents, different baseline approaches to improve stability, and activations. The key takeaways of this experiment are that (1) the CoAN+REINFORCE approach generally underperforms backprop, (2) baselines to improve stability don't seem to help, and (3) the primary reason for the poor performance is the non-deterministic behavior of the other agents (as indicated by having a single agent optimizing all layers performs the best by far). \n\nMotivated by this, the paper presents an off-policy Q learning approach to training cooperative agents, which uses bootstrapping and trains by assuming the best action is taken at the next step. While the paper shows this improves over on-policy SARSA, it still underperforms REINFORCE, which underperforms backprop. \n\nLastly, the paper shows that in a continual learning setting the CoAN+REINFORCE approach can be more robust than backprop. ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n- The paper studies a fascinating topic in using RL to train networks, and the abstract/introduction does a great job in motivating the potential for local/asynchronous training and the possibility of using it for computationally efficient and more general training.\n- While I am less familiar with the prior work in this area, the related work effectively summarized prior approaches and seemed thorough.\n- The empirical study shown in this work on CoANs+REINFORCE is interesting, and sheds some light on what makes the problem challenging (even if the results are still worse than backprop)\n- Similarly, the formulation of using off-policy Q learning in the CoAN framework to address the issue of is a creative and to the best of my knowledge novel way of approaching the problem. \n- The experiment in the continual learning setting is interesting and suggests that in the case of highly correlated data points using the CoAN framework may be more robust.\n\n*Weaknesses*\n\n(1) The clarity and presentation of the empirical study in Section 3 could be improved. There is a lot going on, and the takeaways are currently disorganized. The main takeaway seems to be that the stochastic behavior of different agents is the main source of poor performance from the plot comparing C1-C1-C1 vs C3, but its difficult to follow within all of the other experiments.\nmore specifically:\n- Figure 1 is difficult to follow, particularly with one grid (despite (b) and (c) being completely different experiments). I'd suggest splitting this up if possible and then having more informative captions/legends which explain what the plot is showing. \n- The explanation of the \"baselines\" was very rushed, with just a single sentence each. Without more explanation it is not clear what exactly to take away from Figure 1(a) besides that CoAN+REINFORCE is worse than backprop. Also the baselines seem to be using extra additional parameters? Overall it was not clear what the baselines were meant to show/what the takeaways are.\n- I also found the takeaways from Section 3.4 to be unclear - it makes sense that the straight-through optimizer performs poorly, but does the discretization of nodes actually help CoAN+REINFORCE? From the plots in Fig 1(a) vs Fig 2 (a) they look pretty similar.\n\n(2) I found the Q Learning for CoANs to be an interesting and well motivated approach. However it does seem that the critics need an additional network during training which is optimized separately which seems to be a strong limitation - essentially requiring multiple other networks to train the original network. Even with this extra model capacity it also appears to underperform the initial REINFORCE approach, so its not clear how promising this approach is.\n\n(3) While the continual learning experiment setup is interesting, the CoAN+REINFORCE only appears better in the most extreme cases of correlation [0.98, 1], and its not clear if these results hold generally. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No discussion of broader/societal impact, though not sure how much is needed for this work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "G_ZgXRphe0j", "reviewer": "Reviewer_brEu", "summary": "- This paper revisits an interesting (and truly under-explored) idea from the reinforcement-learning literature: coagent networks [43,44]. The original work by Philip Thomas considered endowing the standard agent, that maps from states to actions, with a richer structure whereby the state can be mapped by multiple coagents both in parallel and in sequence yielding various intermediate outputs (coagent actions) before ultimately rendering a final action to constitute the action output of the overall agent.\n\n- In this paper, the authors consider how the concept of coagent networks may be used to ground neural network training as a finite-horizon reinforcement-learning problem, thereby offering a local, asynchronous alternative to the traditional and widespread method of backpropagation.\n\n- The paper offers a formulation of neural network training as a finite horizon POMDP, where the partial observability arises only to accommodate the non-Markovian nature of the terminal reward signal that must depend on the initial input to the network. While the formulation allows the authors to employ policy-gradient techniques for local, asynchoronous optimization of the network, they highlight shortcomings of using the standard REINFORCE estimator, which align with early findings by [44]. \n\n- On the path to resolving these issues, the authors present numerous experiments that tease apart various hypotheses to diagnose the failures of coagent network training relative to standard backprop. While unable to outperform backprop in the standard supervised learning setting, the authors not only provide ample diagnosis of the failure but also conclude their empirical investigation with a continual learning setup that highlights the strength of using local, adaptive policies for neural network training over backprop. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper scores very highly on significance, quality, and clarity. I commend the authors for the writing style and structure of the paper which gives the reader a sense of the authors' actual thought process when designing each experiment in light of observed results and falsified hypotheses (for instance, L225-233); this is often something valuable that goes unwritten in many research papers. On the axis of clarity, I'm not especially familiar with the literature on alternatives to backpropagation, however the authors do seem to offer a detailed account of prior work in Section 1.1.\n\nI think the meticulous empirical investigation in this paper is something to be excited about as it opens up numerous possibilities for bringing recent advances in RL theory and practice to bear on the problem of efficient, adaptive neural network training. Even though the authors have not succeeded in providing a coagent network setup that outperforms backprop in the standard supervised learning setting, the continual learning experiments are compelling and this paper on the whole will initiate conversations and spark interest from the RL community to help address the outstanding performance gap.\n\nAs a minor point, I did notice that the authors typically refer to the problem formulation as a finite-horizon MDP throughout the text whereas the actual derivation itself specifies the need for a POMDP setup (L144-152). I wonder if perhaps this is unnecessary and the authors might simply consider a non-stationary state space and transition function so that, in the H-1 stage, the state space can be augmented to include the input and all transitions moving from step H-1 to H must fold the input back into the state, leaving it available for the standard Markov reward computation. Another possibility might be to treat the input data as a goal and model training over an entire dataset as a multi-task RL problem [5].\n\nI think there are various pieces of the reinforcement-learning literature that may be useful to think about in future work that builds off the problem formulation presented in this paper such as recent advances in provably-efficient policy gradient methods [1,2], information-directed exploration techniques based on Bayesian reinforcement learning and posterior-sampling methods [3,4], and much more. That said, the authors seem well-positioned (if not, best-positioned) to set the stage for these kinds of discussions; a section in the appendix that outlines some of the enticing but loftier possibilities could be really thought-provoking and interesting.\n\n===== Post-rebuttal =====\n\nWhile I'm still excited by this paper, the comments from the other reviewers certainly suggest that the presentation and clarity could be improved. I have looked at work on conjugate MDPs and policy co-agent networks in the past and so, maybe, the trajectory of the paper unfolded a bit better for me when reading. Nevertheless, for this weakness in clarity alone, I'll reduce my score down to a 7 with the hopes that the authors incorporate suggestions from the other reviewers to boost the presentation for a broader RL audience in a camera-ready version. \n\nGiven the novelty of the problem formulation and the wide range of possibilities that future work might explore, I'm far less concerned by the significance issues raised by the other reviewers (performance relative to backprop and more parameters overall for the critic) and continue to advocate for accepting the paper.\n\n[1] Agarwal, Alekh, Sham M. Kakade, Jason D. Lee, and Gaurav Mahajan. \"On the theory of policy gradient methods: Optimality, approximation, and distribution shift.\" Journal of Machine Learning Research 22, no. 98 (2021): 1-76.\n\n[2] Agarwal, Alekh, Mikael Henaff, Sham Kakade, and Wen Sun. \"Pc-pg: Policy cover directed exploration for provable policy gradient learning.\" arXiv preprint arXiv:2007.08459 (2020).\n\n[3] Osband, Ian, Benjamin Van Roy, Daniel J. Russo, and Zheng Wen. \"Deep Exploration via Randomized Value Functions.\" J. Mach. Learn. Res. 20, no. 124 (2019): 1-62.\n\n[4] Russo, Daniel, and Benjamin Van Roy. \"Learning to optimize via information-directed sampling.\" Advances in Neural Information Processing Systems 27 (2014): 1583-1591.\n\n[5] Schaul, Tom, Daniel Horgan, Karol Gregor, and David Silver. \"Universal value function approximators.\" In International conference on machine learning, pp. 1312-1320. PMLR, 2015.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors are quite upfront in their discussion of the limitations in the proposed coagent network approach and adequately highlight paths forward for future work to consider in bypassing these obstacles.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper started out with a strong accept and 3 weak rejects, but then converged to 7-6-6-5 (where reviewer with 5 has a short review and is less engaged than other 3 reviewers). I recommend the weak acceptance for the paper, primarily because the novelty/insights of the paper outweigh the immaturity of presentation/lack of pragmatic result (to outperform backprop). The formulation could be impactful for further research, and this paper lays out sufficient foundations for it. However, all reviewers agree that the presentation clarity must be substantially improved, and therefore the authors must follow through and incorporate the reviewer suggestions for the final version. 
Decision: Accept (Poster)