Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This work combines two existing approaches: representing a policy based on its behavior in a set of probing states and finding a successful policy via a critic that estimates the return of any input policy.

The ability to automatically learn probing states is shown via MNIST.
The performance on a few continuous control tasks is demonstrated.
The ability to use a trained critic to find a successful linear policy is also shown.

strength_and_weaknesses: This work identifies a promising combination of two existing methods.
If probing states are a more efficient representation for complex policies, then this method is of interest.

The authors claim that "Flattening the policy parameters ... is difficult to scale to larger policies," but do not show the new method scaling.
Unfortunately, the proposed method is not shown to scale to larger domains. Empirical evaluation is required to confirm favorable scaling, but all of the domains considered can be solved with simple policies.
I am skeptical that this method scales better than using the policy parameters: a strength of policy gradient methods is that the policy complexity generally scales more slowly than the environment complexity. However, this method requires coverage of the state space, so the number of required crucial states may scale quickly.
The key experiment is shown in the appendix (Fig 10) and is quite limited in scope. It specifically uses the environments where few probing states are needed to learn a policy.

Minor Comments:
- The "Policy fingerprinting" paragraph should be divided to more clearly partition novel contributions from existing work.
- By changing the rewards for Hopper, Walker, and Ant, those environments are no longer being solved (only modified versions of them). Likewise, gamma is part of the MDP; reporting total return (gamma=1) when gamma is 0.99 for only some evaluated methods is incorrect.
- Sec 4.3: "trough" -> "through"
- The limitation of the proposed method to deterministic policies is used to motivate baseline selection, but this limitation is a choice rather than part of the problem setup. To use this line of reasoning, one should also express why deterministic policies are advantageous.

Questions:
- How were domains selected for evaluation? Why are only two environments used for Fig 10?
- What is the behavior of your method with stochastic policies?
- Can your approach perform well when the policies have different architectures during training?
- Can vanilla PSSVF be used with weighted sampling?
clarity,_quality,_novelty_and_reproducibility: The combination of methods is novel, but the utility is not demonstrated.

In addition, the experiments are oddly limited:
- only continuous control problems are used for main experiments
- only deterministic policies are used for main experiments
- only environments where few crucial states are needed are used for evaluation
To support the claim that probing states are a better representation, thorough comparisons between different representations should be performed.
In particular, to demonstrate that probing states scale better, evaluations should be performed with more complex environments.

The prioritization of content inclusion needs to be improved. Key experiments are in the appendix while lots of content in the main body can be cut or moved to the appendix.
Note that "reviewers are not required to read the appendix" (ICLR 2023 CFP). I have still read the appendix, but I urge the authors to move key experiments to the main body.
Examples:
- Sec 2 and 3 contain lots of non-novel content that is not needed within this work.
- Likewise, "a demonstration that fingerprinting can learn interesting states in MNIST" is of limited interest (no hypothesis is being tested; no baselines are used for comparison).
- Also, learning a near-optimal linear policy does not show that this method has an advantage over directly using policy parameters (i.e., this ability is due to PBVF).
- Evaluating the contribution of policy fingerprinting is crucial, yet it is in the appendix.
summary_of_the_review: The proposed method is motivated by the difficulty scaling a policy-parameter-based approach. However, this benefit is not supported.
Additionally, the overall experiments are limited without sufficient justification.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: 
This paper proposes using the policy evaluation networks architecture
(policy fingerprinting) with the algorithm structure of parameter-based
value functions to learn a single value of the start state that
conditions on the policy representation. This value function is learned
through online interaction, like parameter-based value functions, but
scales to larger policies because of policy fingerprinting. The
eponymous finding is that, for some mujoco environments, the number of
probing states for policy fingerprinting is surprisingly low while
facilitating zero-shot policy learning on different architectures.


strength_and_weaknesses: # Strengths

-   The main result is very interesting and surprising. The result being
    that policies in mujoco environments can be represented by observing
    the actions at a small number of states. It is further surprising
    that a strong policy can be learned by regressing on the actions at
    those few states.

-   While there are a few clarity issues, which I outline in the
    detailed comments below, the overall paper is well-structured and
    easy-to-follow. Not only is the motivation clear, the contribution
    is clear as well.

# Weaknesses

-   The main weakness is that the novelty is incremental.
    Algorithmically, there is little novelty: the proposed algorithm is
    a combination of the architecture of policy evaluation networks and
    online algorithm structure of parameter-based value functions. The
    eponymous finding, that policy evaluation and improvement can be
    done by identifying few but crucial states, is limited to only two
    mujoco environments. While the experiments encompass more than this
    finding, this seems to be the major contribution.

clarity,_quality,_novelty_and_reproducibility: # Detailed Comments (Clarity, Quality, Novetly, Reproducibility)

-   Section 2 (Stochastic vs deterministic): Relatively minor but, I do
    not think these two cases should be treated separately, as the
    objective written in equation 1 is still the objective being
    maximized. The policy-gradient theorem merely provides an
    alternative update for the policy directly. The deterministic
    objective can still be thought of as an integral, where the
    distribution is a dirac delta on the deterministic action.

-   Section 3 (Probing states and policy visitation distributions): The
    idea of having a single value function for many policies is
    appealing. The parameter representation is a global description of
    the policy. But, the probing states are local and depend on the
    policy and the visitiation distribution it induces. For example, two
    deterministic policies travelling in separate directions will never
    visit the same states, and hence the probing states may not be
    shareable amongst these two policies.

    Is there a theoretical justification or intuition for evaluating all
    the policies on one distribution of probing states (even if these
    states are learned)?

-   Section 3 (Policy Improvement and Probing states): Because the
    probing states are learned, it may be the case that the probing
    states do not correspond to any real state in the environment.
    Improvements on these probing states benefit the policy globally
    only through generalization. Returning to the "two direction MDP"
    example from my earlier point, the action on the learned probing
    state could be sufficient for evaluation/prediction and yet may not
    provide a useful gradient.

-   Section 4.1 (Offline policy improvement): It is quite surprising
    that, after constraining the loss to be 12\%, the value function is
    able to extrapolate and provide a gradient that improves the CNN to
    65\%. MNIST, however, is a relatively simple dataset which can be
    classified with a linear decision boundary. I wonder if similar
    results hold on harder datsets, such as omniglot, fashion MNIST and
    CIFAR.

# Minor Comments

-   Section 2: Markovianity -> Markov Property
-   Section 4.3: "trough" -> through


summary_of_the_review: It is difficult to rate this submission because, while lacking in
novelty, the eponymous finding is rather surprising and interesting. The
lack of novelty in the algorithm is unavoidable, as it builds on two
distinct works that are quite similar. However, I think the experiments
have potential to provide more insight. While the results are mostly
positive, the submission could be strengthened by outlining the limits
the approach in negative results. For example, is it more or less
difficult to learn to predict the losses when the dataset is more
complicated (omniglot, fashion MNIST, CIFAR). Does this difficulty
translate to difficulty to extrapolate to new architectures, or to
zero-shot train an architecture to a performance level beyond what was
seen? There is plenty of opportunity to contextualize the very
interesting main finding. As the submission currently stands, I am
leaning towards weak reject. I am open to increasing my score of some of
these points are addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors get the parameter-based state-value function (PSVF) Faccio et al., 2021 to work. The motivation is simple, if there's a value function $V:\mathcal S\times\Theta\to\mathbb R$ which maps both a state and the parameters of a policy to an accurate cumulative discounted reward estimate, then the policy can be improved by maximizing $V(s,\theta)$ w.r.t. $\theta$. The only problem is that for any class of policies to be useful, the parameter space of that class of polices $\Theta$ is generally quite large (thousands of dimensions) making learning the PSVF V impractical.

The authors propose to map this large $\mathcal S\times\theta$ to a smaller space with two simple tricks. First, they don't worry about the discounted return will be from a specific state, peruse a general policy evaluation framework to learn the sum of discounted rewards from an average starting state. This isn't novel. The novel and significant step they take is to learn K "probing states" $s_1,\dots,s_K$ and instead of learning the PSVF they estimate (this notation only works for deterministic policies, but the idea works for certain stochastic policies as well) the value of a policy with $V_\phi(\pi_\theta(s_1),\dots,\pi_\theta(s_K))$. Since $K$ is small (generally 200 or less) and the action space is also small, this addresses the curse of dimensionality.

The authors then do two sets of experiments, one on MNIST and the other on MuJoCo outlining interesting characteristics of their PSSVF method, showing that the probing states learned are meaningful and that the learned value estimation is useful for constructing good policies.





strength_and_weaknesses: I'll compress "Strength And Weaknesses" and "Clarity, Quality, Novelty And Reproducibility" into one review, as there's a lot of overlap.

### Strengths:
- the paper has a clear goal which is quite relevant: learn a single value function for evaluating any policy, and the authors clearly explain how they go about working to this goal, compare to past work on how this goal was achieved, and what they do different, and why this leads to better results.
- the paper's notation is generally clear, well introduced and easy to follow
- The experiments show what the method is doing, not just that it gets good results but that the probing states are significant, and that a value function learned even on random policies can be used to generate a good policy.
- I like that the authors don't claim their method is the best, but compare against more sample efficient methods like SAC.

### Major Concerns:

My only major concern is that the authors seem to claim to do more than they actually achieve. The claim is to "learn a single value function for evaluating...any policy" But they don't do this, they learn an estimate $J(\theta)$ of the expected return. Worse, they really dress the paper up to make it look like they learn a value function, using $V(\theta)$ notation for example, and discussing value functions and Q-functions at length in the introduction. While I'm sure their methods could be modified to learn an actual parameter-based state-value $V(s,\theta)$ I bet it's tricky (since most of the time the estimate of the value function depends on an estimate of the value function).

So I'd really wish the author's are honest with their readers and say something like "in order to focus on the main point, the probing state method and not get distracted by the complexities of learning a value function, we focus on the easier task of learning estimate $J(\theta)$ of the expected return on finite horizon MDPs. And drop anything that distracts from this simpler yet significant goal.

The authors discuss a tiny bit the step to learning a real value function at the top of page 7, but it's a bit of a farce, since they only learn the value function over initial states, if I remember correctly in MuJoCo initial states are all quite similar, so it's unsurprising that the authors state "The results were very similar to those we presented in this section"

For this reason, I have to say on the correctness that "Several of the paper’s claims are incorrect or not well-supported," but I don't see any reason why this can't be fixed.

### Minor Concerns:
- I don't find Figure 1 helpful at all. Why are some lines solid, why are some dashed? What's $U_\phi$? The algorithm has a replay buffer (or storage of policy parameters / rewards) that's not in Figure 1. Either have a helpful figure or no figure at all but not a figure for figure's sake.
- I think that Eq. 2 is incorrect, but since I think it's a distraction and should be dropped I won't go into detail.
- in eq. 4, $\pi_\theta(s_1)$ is a distribution over actions. You discussed "parameters of the output distribution of the policy in such states" but left that out in eq. 4. For eq. 4 to be correct, this mapping from $\theta$ to distribution parameterization must be included.
- Please also tell the readers that the estimation of $V(\theta)$ only works since in MuJoCo the starting states are all very similar, and the MNIST experiment is stateless.


clarity,_quality,_novelty_and_reproducibility: see above for clarity, quality and novelty.

The experimental setup is nicely explained in the appendix, and I think reproducible. 
summary_of_the_review: Overall, when reading the paper I felt I learned a significant new tool, one that I feel others in the RL community should learn
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper "General Policy Evaluation and Improvement by Learning to Identify Few But Crucial States" proposes to consider fingerprints based on actions distributions on states, rather than using all policy parameters, for building value functions that generalize over all possible policies. Such policy evaluation functions are very useful for discovering efficient policies, by gradient ascent of policy parameters through the value function, as they do not require interactions to improve policies whenever the value function is general enough. Authors show interesting results.  
strength_and_weaknesses: Strengths:
   - Interesting results : only few states are enough to get useful value functions and policy improvements in many classical settings
   - Well written paper

Weakness: 
    - I feel the approach is a very straightforward extension of [Harb et al., 2020], as the only novely comes down to simply alternating the evaluation and improvement algorithms from that paper (except a very minor difference on the way of considering rewards - as a regression target rather than classification in discretized bins). 
    - Thus the interest of the paper is mainly experimental. However I feel that not enough analysis is given to well understand the dynamics of the proposal.   

clarity,_quality,_novelty_and_reproducibility: Clearly written paper, but novelty is limited
summary_of_the_review: I already gave my main concerns above, I have now some questions for the authors: 
    - Is there no catastrophic forgetting during learning, since V has to embed every encountered policies ? Is there not a risk of oscillation between two parameters areas ?  
    - From my point of view, the experimental part miss a study of probing states dynamics. It would be veryinsightfull to understand how they behave during learning. Do they first collapse in a single state while the value is not well learned for spreading on more useful areas afterwards ? 
   - As PVN is a very related approach, I would have expected to have results of this approach as a baseline, maybe with a version that alternate evaluation and policy improvement
    - Did authors experiment a version with reward discretization as in PVN ?  
   - Algorithm1 only learns from feedback over full trajectories. Thus the variance must be high. How would it perform on more difficult stochastic environments ? I suspect that considered tasks are rather simple and this kind of approach would be difficultly transferred on more complex environments. 

In summary, a very easy to read and interesting paper, but from my point of view the proposal is not enough innovative for ICLR, and the experimental analysis not enough insightful.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Strengths: The main result is that policies in mujoco environments can be represented by observing the actions at a small number of states. It is surprising that a strong policy can be learned by regressing on the actions at those few states. If probing states are a more efficient representation for complex policies, then this method is of interest.

Weaknesses: The authors but do not show that the new method scales.  All of the domains considered can be solved with simple policies. IA reviewer is skeptical that this method scales better than using the policy parameters. The key experiment is shown in the appendix (Fig 10) and is quite limited in scope. It specifically uses the environments where few probing states are needed to learn a policy.

Also, the novelty is incremental. The proposed algorithm is a combination of the architecture of policy evaluation networks and online algorithm structure of parameter-based value functions.

Overall the reviewers did find sufficient merits in this paper.
justification_for_why_not_higher_score: See the weaknesses listed above.
justification_for_why_not_lower_score: This is the lowest score.
Decision: Reject