Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: ### Summary:
The paper proposes passing the parameters of a policy to the value function attempting to learn estimates of the return for that policy. This allows the value function to generalize across policies and estimate values for arbitrary policies. The paper derives several algorithms for various objectives and value functions, and empirically investigates the deterministic versions.

### Pros:
- Several new algorithms are proposed
- The new algorithms can generalize across policies
- The new algorithms can estimate the value of unseen policies

### Cons:
- Only the deterministic algorithms are empirically investigated
- Computation and memory cost seem quite high (the critic takes all of the actor’s parameters as arguments)
- Empirical results seem mixed

### Decision
I recommend accepting the paper for publication.

The paper investigates a simple, interesting, original idea—including the actor’s parameters as inputs to the critic—fairly thoroughly. Several actor-critic algorithms are derived using expressions for the gradient of various performance measures obtained by including the actor’s parameters as inputs to the critic.

The benefits of doing this are illustrated by some experiments, and the deterministic versions of the new methods are compared with reasonable competitors (DDPG and ARS) in other experiments. Unfortunately the results seem somewhat limited by the number of runs that can be conducted by parameterizing the policies and value functions as neural networks and experimenting on the chosen environments. Overall the empirical results seem mixed; in many environments it’s fine to just disregard the second part of the gradient that is dropped in DDPG and computed by PVFs. However, that’s not the fault of the new algorithms, and there are some environments where not dropping the second part of the gradient is helpful.

The paper is clearly written for the most part, with the exception of some parts of the related work that are overly terse (i.e., the connection with UVFAs could be expanded). Other parts of the related work seem frankly unrelated (i.e., predicting gradients of RNNs from their inputs in the 90s, and mapping weights of CNNs to their accuracy), and I would recommend removing them in favour of moving the more detailed comparison of PENs and PVFs into the main paper.

### Miscellaneous comments:
- Grammatical error in the final sentence of the abstract: “Their performance is comparable to the one of state-of-the-art methods”
- “In practice, like in standard actor-critic algorithms, we use a noisy version of the current learned policy in order to act in the environment and collect data” This should probably read standard deterministic actor-critic algorithms.
- I was disappointed to see that only the deterministic algorithms were implemented and analysed. Even if the stochastic versions of the algorithm are only demonstrated in a simple linear setting, that would be better than just not investigating them at all.
- The paper doesn’t mention related work that fixes the Off-PAC policy gradient theorem, which gives an expression for the true gradient of the off-policy objective without requiring PVFs (Imani 2018).
- Passing the actor’s parameters to the critic seems to necessarily break the requirement of compatible features for the actor to follow the true gradient of performance (Sutton 2000). It might be good to mention this.

### References:
1. Imani, E., Graves, E., & White, M. (2018). An off-policy policy gradient theorem using emphatic weightings. In Advances in Neural Information Processing Systems (pp. 96-106).
2. Sutton, R. S., McAllester, D. A., Singh, S. P., & Mansour, Y. (2000). Policy gradient methods for reinforcement learning with function approximation. In Advances in neural information processing systems (pp. 1057-1063).

review_title: Interesting idea that is well-investigated

final_score: rating: 7: Good paper, accept
review_content: On page 2, in the background section: the discounted state distribution, what you wrote is not a distribution (doesn't sum to 1). In order to define this $d^{\pi_\theta}$ properly, you can multiply everything by $1-\gamma$. The interpretation is that you "reset" in your initial distribution $\mu_0$ with probability $1 - \gamma$ at every step, or continue in the discounted stationary distribution with probability $\gamma$.

In think that theorem 3.1 is incorrect. I think that this is meant to describe an off-policy setting where we are collecting data from $\pi_b$ but want the policy gradient for $\pi_\theta$. In this case, the importance sampling weight should be $\frac{d_\theta(s,a)}{d_b(s,a)}$ not $\frac{\pi_\theta(a|s)}{\pi_b(a|s)}$ (where $d_b$ is the discounted stationary distribution, see above comment too). Equation 9 follows from the chain rule (because the Q function now depends on $\theta$ explicitly) using the off-policy formulation in Degris (2012), which is incorrect.

Notes:
- PVF: to me this acronym is strongly synonymous with Mahadevan's proto-value functions (PVFs), circa 2007. How about "PBVF" instead? Maybe I'm old

> we optimize for the undiscounted objective
this should be reflected in your notation and problem formulation

> can be used only for episodic tasks
it doesn't have to. See "regenerative method" in Monte Carlo estimation literature 
review_title: Interesting connection to DPG, few technical errors

final_score: rating: 6: Marginally above acceptance threshold
review_content: — idea:
	
A new class of value functions is introduced where the value function takes the parameters of the policy as input, in addition to its common inputs (state or state-action). The proposed type of value functions, PVFs, are also useful for off-policy learning and generalizing over policies while the common value functions lost their information about previous policies.

— comments:
   
1- Regarding the first algorithm, PSSVF, until converging, the data that is stored in the replay buffer does not correspond to a "reasonable" policy, unless having a prioritized replay buffer. I am also concerned about it being over fitted to the early policies and not being able to  overcome this. I see in the experiments that using PSSVF, policy is converged but am not convinced about it.
    
2- Another concern of mine wrt the proposed PVFs is about their sample efficiency. It would be interesting to see a comparison between DDPG and PVF based methods on their sample efficiency.
    
3- In the experiments section (4.2), expect for a few cases, ARS is either the best one or does not differ significantly from PVF based methods. Having this in mind, my question is that have you tried to find the best set of hyperparameters for ARS and DDPG as well as your proposed method? If the answer is no, I would like to see that experiments where ARS and DDPG have their best set of hyperparameters.
    
4- I am also interested in seeing results for deep policy zero shot learning. In section 4.3, authors just mention: "When using deep policies, we obtained similar results only for the simplest environments." which is not convincing without showing results.
    
5- As mentioned in the last part of the paper, the proposed method hugely suffers from the curse of dimensionality. However, as an initial step, PVFs seem interesting and could be beneficial in terms of learning generalized value functions.

-- minor issues:
    
In the first line of the paragraph above the experiments section (4), starting with "Algorithm 4 (Appendix) uses an ...", there is a redundant "and". One of them should be removed.



Overall, I liked the idea presented in the paper and would like to see what their next step would be. But the current version of the paper could benefit from more in depth experiments. I believe the most important weakness of the paper lies in the experiment section. It can be much richer and more insightful.



[1] Sutton, Richard S., et al. "Horde: A scalable real-time architecture for learning knowledge from unsupervised sensorimotor interaction." The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 2. 2011.
review_title: Interesting in terms of idea, but unclear advantage over common approaches

final_score: rating: 7: Good paper, accept
review_content: **Update**

I have updated my score to 7.
One of the points that was not explained in the original paper was that (ignoring function approximation effects) an optimal solution for $J_b$ (the OffPAC objective) will be optimal also for the original off-policy RL objective $J$ (i.e. estimating the on-policy objective in an unbiased manner from off-policy data). From this point of view, I agree that optimizing $J_b$ directly is an interesting question, despite the fact that the exact gradient for $J_b$ may be less similar to the gradient of $J$ compared to the usually used approximate gradient of $J_b$ that drops the $\nabla_\theta Q$ term. It still remains unclear which of the two methods has a theoretical advantage over the other in the function approximation setting (in terms of optimizing for $J$); however, because it is unclear, it is interesting to evaluate the method proposed here and to perform experiments as done in the paper to try to find out which method performs better.

The results were mixed; however, the evaluation is fairly thorough and some potential advantages of the new methods such as generalization in the $\theta$ space and zero-shot learning were explained.

The discussion in the paper is much improved compared to the original version. Also, additional ablation studies such as testing what happens when the $\nabla_\theta Q$ term is dropped were added (when $Q$ includes $\theta$ as an input). Moreover, LQR experiments for $Q(s,a,\theta)$ and $V(s,\theta)$ were added in the appendix (the results here do not give as good a match as the $V(\theta)$ formulation gave, but they are reasonable).

______________________________________________________________
1. Summarize what the paper claims to contribute. Be positive and
generous.

They propose to include the policy parameters as an input
to the value function, so that the value function could generalize
across different policies (there are 2 other concurrent works with a similar
idea, one they have cited and discussed "Policy evaluation networks"
https://arxiv.org/abs/2002.11833, another is submitted to ICLR2021 on
openreview https://openreview.net/forum?id=V4AVDoFtVM).
They put the policy parameters theta as an input to the value function
in 3 cases $V(\theta)$ (PSSVF), $V(s, \theta)$ (PSVF), and $Q(s,a,\theta)$ (PAVF).
They propose new policy gradient theorems for the $V(s,\theta)$ and
$Q(s,a,\theta)$ cases (but I believe these to be theoretically flawed).

They perform experiments testing $V(\theta)$ in 2 cases: 4.1) (sanity check
experiment) visualizing and testing for correctness on an LQR task,
4.3) zero-shot learning: after training a policy pi using the $V(\theta)$ method,
a new policy is reinitialized pi_new and trained from scratch using only the
trained $V(\theta)$ without interacting with the environment. The interesting
bit was that $\pi_{new}$ managed to outperform the learned policy during
data collection $\pi$ (this implies that the $V(\theta)$ function managed to
generalize. It would have been nice to, in addition to $\pi_{new}$, also
see whether $\pi$ could have been improved by just continuing to optimize
it without interacting with the environment, but this was not done).

They tested $V(\theta)$, $V(s,\theta)$ and $Q(s,a,\theta)$ on MuJoCo tasks
compared to augmented random search (this is similar to evolution
strategies) and to deep deterministic policy gradients (DDPG). And
the performance did not change much, and sometimes all the new methods
failed when DDPG worked (on the reacher task).

The final experiment 4.4 was for offline learning with fragmented
behaviors, i.e. they do not observe full episode data for a fixed
theta, which makes it impossible to learn $V(\theta)$ directly, but
$V(s,\theta)$ can be learned by TD methods (also note that the data is
collected from a different behavior policy). Then they test a similar
zero-shot learning procedure as they did for $V(\theta)$ at different
stages of the learning (but as far as I understood, for $V(s,\theta)$ they sampled
data from the replay buffer when training the policy (thus not fully without
interacting with the data). Perhaps the authors can clarify this), and
show that the newly learned policy can outperform the behavior policy,
thus demonstrating the generalizability of the method.


2. List strong and weak points of the paper. Be as comprehensive as possible.

\+ The experiment on zero-shot learning is nice to show that the $V(\theta)$
function can generalize.
\+ The paper is clearly written.
\+ They discuss a lot of related work.
\+ The experimental methodology seemed mostly good and honest, and
was explained in detail in the appendix (some nice points: They include
a sensitivity analysis showing quantiles of the performance.
Also the final best chosen hyperparameters were evaluated with
20 new seeds, separate from the 5 seeds used during hyperparameter tuning).

\- The new policy gradient theorems seemed flawed. Also some discussion
around off-policy learning seemed incomplete.
\- The methods were not shown to experimentally lead to major gains.
\- One of the difficulties with searching in parameter space is how
to deal with large parameter spaces. The two concurrent works considering
$V(\theta)$ proposed solutions to this issue by embedding the policy into
a smaller space. In the current work no solution is proposed. The
experiments on zero-shot learning using $V(\theta)$ were only good with
low-dimensional linear policies.
\- A sanity check experiment on LQR was performed for only $V(\theta)$ (which was
the only one for which the gradient was theoretically sound); it would
have been good to do similar experiments for the other ones.
\- I would expect $V(s,\theta)$ to outperform $V(\theta)$ due to using the
state information, but this did not appear to be the case.


3. Clearly state your recommendation (accept or reject) with one or two
key reasons for this choice.

I recommend rejecting the paper due to the theoretical flaws in the newly
proposed policy gradient theorems using $V(s,\theta)$ and $Q(s,a,\theta)$. Also,
the practical advantages of using $V(s,\theta)$ and $Q(s,a,\theta)$ were not shown.


4. Provide supporting arguments for your recommendation.

The theoretical issues in this paper start in equation 1. They write:
"... we can express the maximization of the expected cumulative reward
in terms of the state-value function:"

$J(\pi_\theta) = \int d^{\pi}(s) V(s) ds,$  (in the paper)

where $d(s)$ is the discounted state visitation distribution. However, this
is not the RL objective. The RL objective would be

$J(\pi_\theta) = \int d^{\pi}(s) R(s) ds.$    (what it should actually be)

The authors probably took their objective from the work by
Degris et al (2012, https://arxiv.org/pdf/1205.4839.pdf);
however, in Degris'12, $d(s)$ is _not_ the discounted
state visitation distribution. It is the limiting distribution as
$t \to \infty$, which is a stationary distribution. When $d^{\pi}(s)$ is stationary,
then the two objectives become equivalent: $d(s)$ does not change
from one time step to the next, so the difference between the objectives
will be just a $1/(1-\gamma)$ constant factor. Putting aside this issue,
probably the limiting distribution formulation is not realistic as most
RL researchers consider the episodic setting, so using a discounted
state visitation distribution is probably better. However, the newly
proposed policy gradient theorems do not appear sound for the true RL
objective using $R(s)$.

Next, they replace the distribution $d^{\pi}(s)$ with a
distribution $d^{\pi_b}(s)$ gathered using a behavioral policy (so they are
working off-policy). However, they do not apply an importance weighting
correction for the distribution shift, and just ignore the importance
weights (Note that this is also done by Silver et al (2014) in deterministic
policy gradients, and by Lillicrap et al (2015) in DDPG, so it is not that
strange per se, as long as it gives better practical performance. However,
it should at least be acknowledged that the importance weights are being
ignored). Note that they still apply an importance weight on the actions
($\pi(a|s)/\pi_b(a|s)$) once the state is sampled from the data buffer, however,
this does not correct for the distribution shift from $d^{\pi}$ to $d^{\pi_b}$,
so the policy gradient computed using such a method will necessarily be biased.
For example, see the following works for examples that try to deal with the
distribution shift problem:
Munos et al (2016, https://arxiv.org/abs/1606.02647),
Wang et al (2016, https://arxiv.org/abs/1611.01224),
Gruslys et al (2017, https://arxiv.org/abs/1704.04651)

Putting aside the issue of whether ignoring the distribution shift is OK,
the main issues are the new policy gradient theorems derived from this
formulation. Both the $V(s,\theta)$ as well as $Q(a,s,\theta)$ formulations appear
flawed:
In the $V(s,\theta)$ case they propose the policy gradient:

$\nabla_\theta J(\theta) = \int d^{\pi_b}(s) dV(s,\theta)/d\theta ~~ds$ in equation 8.

However, the true policy gradient is:
$\nabla_\theta J(\theta) = \int \mu(s) dV(s, \theta)/d\theta ~~ds,$
where $\mu(s)$ is the start-state distribution. Actually they wrote
this also in equation 7, when they considered the $V(\theta)$ formulation,
but for some reason sampled from $d(s)$ instead for $V(s, \theta)$ when computing
the policy gradient in the $V(s,\theta)$ formulation.

In the $Q(a,s,\theta)$ formulation, they add an extra $dQ/d\theta$ term to
the policy gradient. Their motivation is the following:

$\nabla_\theta J(\theta) = \int d^{\pi_b} (dQ(a=\pi(s,\theta),s,\theta)/d\theta) dads$
  $   	= \int d^{\pi_b} dQ(a,s,\theta)/da*da/d\theta + dQ(a,s,\theta)/d\theta~~ dads$

However, this derivation stems from the flawed definition of J that is
not maximizing the sum of rewards over the trajectory distribution, but
maximizing some other objective that sums the value functions at all states
in the trajectory distribution. My strongest argument for why the original
off-policy derivations by Degris et al and Silver et al are less flawed is
the following:
If we are on-policy, i.e. $\pi_b = \pi$ and $d^{\pi_b} = d^{\pi}$ we would want
the off-policy policy gradient theorem to be unbiased, hence it should
revert to the standard policy gradient theorem. In the formulations
of Degris and Silver, this is indeed the case, and these theorems would
be unbiased in the on-policy setting. The new theorem in the current
paper, on the other hand, would have an extra $dQ/d\theta$ term, which would
bias the gradient. Therefore, I do not see any good theoretical reason to
add this term. Moreover, the practical performance did not improve, so
there is little evidence to suggest it as a heuristic either.

If someone would say that the original policy gradient
theorem requires the $dQ/d\theta$ term, I would urge them to look at the original
proofs---there is no approximation, these theorems are exact for the true
RL objective based on maximizing the rewards over the discounted trajectory
distribution. The intuition is that the remaining $dQ/d\theta$ term for the
remainder of the trajectory from a time-step t is estimated by summing
the $dQ/da\*da/d\theta$ or $Q\*dlog/d\theta$ terms for all the future time-steps.

Another more minor theoretical issue in the paper is that while the
theory considered the discounted state visitation distribution, the
discount factors are not added into the policy gradient in the algorithmic
sections. This omission is common, and tends to work well as a heuristic
(but it should at least be mentioned that such an approximation is made).
See the following papers for more discussion on this:
Nota and Thomas (2020, https://arxiv.org/abs/1906.07073)
Thomas (2014, http://proceedings.mlr.press/v32/thomas14.html)


5. Ask questions you would like answered by the authors to help you clarify your understanding of the paper and provide the additional evidence you need to be confident in your assessment. 

How did the computational times compare? Was there much of an overhead to
using the more complicated critics including theta as an input?


6. Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.

For me to change my assessment, first the theoretical issues should be
fixed or cleared up.

Next, I have some possible suggestions:
1) Test also $V(s,\theta)$ on LQR as well as on zero-shot learning while sampling
s from the initial state distribution $\mu(s)$. This does not require interacting
with the environment (because you never apply any action), and I would consider
it fair in terms of comparing to $V(\theta)$. If the learning from the TD error
is working well, I would expect it to outperform the $V(\theta)$ formulation
in the zero-shot task. 
2) Test the parameter value functions using the standard policy gradients
without adding the $dQ/d\theta$ term. Because you are using $Q(a,s,\theta)$, there
may be some learning to generalize across different policies due to the
theta input, so it may outperform the original policy gradients without
changing the policy gradient theorem. Actually, it would have been better to
perform such experiments as an ablation study from the beginning anyhow.
3) Test $Q(a,s,\theta)$ also on the LQR task to show it's correctness
(for example by sampling s from the initial state distribution and computing
the action). It may also be nice to test it in the zero-shot task as well.
4) Perhaps test combinations of the various gradients, for example taking
the average of the $V(\theta)$ gradient with the policy gradient using $Q$
(i.e. taking the average of two equivalent policy gradients).

If the above points are convincingly done, I may increase to marginal
accept. The current contributions are not enough for me to go higher than
that: taking away the proposed new policy gradients, the main contribution
is to add $\theta$ as an input to $V$ and $Q$, which I think is not enough.
Moreover, the advantage of adding $\theta$ as an input was not shown convincingly
using compelling evidence. Currently the most compelling evidence is the
zero-shot task, which shows that there is some generalization happening in
the $\theta$ space; however, what is missing to me, is a demonstration of how
this additional generalization helps in solving the original task in a more
data-efficient manner. Perhaps interleaving the policy search with longer
sessions of off-line learning (without any interaction) using $dV/d\theta$
to take advantage of the generalization may improve the data-efficiency
and show the advantage of the new method (exaplaining good practices on how
to do this may be a useful contribution). I think it would also be important
to show compelling evidence that including the s input helps in learning
better $V$ and $Q$ functions. Perhaps there are also other ways to better
show the advantage of the method.

Another option may be to change the problem setup, so that the new policy gradient theorems would be more sound. For example, using the original formulation of Degris'12 where $d^{\pi_b}(s)$ is the limiting distribution as $t \to \infty$ would make the new policy gradients correct; however, the standard setup would not correspond to this. One setup that would correspond to this objective is the following: an infinite horizon continuing setting, where the agent is never reset into the initial distribution, but has to continually change the policy to improve. The learning would iterate between running one behavioral policy until it converges to its stationary distribution, then optimizing a new policy while in the off-policy setting, then switching the behavioral policy to this new policy, and repeating the process. In this situation, $d^{\pi_b}(s)$ can be seen as the initial distribution for the new policy, and in this case the new policy gradient theorems would make sense. My previous argument about wanting the policy gradient theorem to be unbiased in the on-policy case would also be satisfied, because if $d(s)$ is stationary then 
the $dQ/da\*da/d\theta$ and $dQ/d\theta$ gradients would differ by only a constant factor.

review_title: Reject for theoretical reasons. (Update: the theoretical issues were cleared up)

Meta-review: The reviewers generally found the idea interesting and the contribution of the paper significant. I agree, I think this is quite a neat idea to investigate, and the paper is written well and is engaging to read.

I would encourage the authors to take into account all of the reviewer suggestions when preparing the camera-ready version. Of particular importance is the name: I think it's bad form to appropriate a name already used in other prior work (proto-value functions, which are very well known in the RL community), so I think it is very important for the final to change the name to something that does not conflict with an existing technique. Obviously this does not affect my evaluation of the paper, but I trust that the authors will address this feedback (I will check the camera-ready).
Decision: Accept (Poster)