Studying the Interplay Between the Actor and Critic Representations in Reinforcement Learning

Authors: Samuel Garcin, Trevor McInroe, Pablo Samuel Castro, Christopher G. Lucas, David Abel, Prakash Panangaden, Stefano V Albrecht
Primary area: reinforcement learning
Keywords: Reinforcement Learning, Representation Learning, Transfer Learning, Information Theory

Abstract:
Extracting relevant information from a stream of high-dimensional observations is a central challenge for deep reinforcement learning agents. Actor-critic algorithms add further complexity to this challenge, as it is often unclear whether the same information will be relevant to both the actor and the critic. To this end, we here explore the principles that underlie effective representations for the actor and for the critic in on-policy algorithms. We focus our study on understanding whether the actor and critic will benefit from separate, rather than shared, representations. Our primary finding is that when separated, the representations for the actor and critic systematically specialise in extracting different types of information from the environment---the actor's representation tends to focus on action-relevant information, while the critic's representation specialises in encoding value and dynamics information. We conduct a rigourous empirical study to understand how different representation learning approaches affect the actor and critic's specialisations and their downstream performance, in terms of sample efficiency and generation capabilities. Finally, we discover that a separated critic plays an important role in exploration and data collection during training. Our code, trained models and data are accessible at https://github.com/francelico/deac-rep.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1730066530683

thread_type:
official_review

<thread_object>
creator: reviewer_r4wb

<content>
summary:
Authors systematically investigate the impact of coupling and decoupling of the representations for actors and critics in on-policy actor-critic framework.

strengths:
1. Theoretically-grounded study with the demonstration that actor and critic should be decoupled as their objectives contradict each other.
2. Good empirical study for confirming claims using various algorithms. When result contradict assumptions, authors test and confirm different hypothesis which could explain the observations. These assumptions reveal novel things for RL community, such as the potentially underestimated role of critic in exploration.
3. Additional study on the effects of different representation learning approaches and when they benefit performance.

weaknesses:
1. Study is only conducted using the on-policy algorithms. It would be interesting to also investigate off-policy methods
2. If I understand correctly, study is performed only on discrete state and action spaces environments. Would the conclusions change if we switch to the continuous environments? It is not clear for me whether the proposed analysis can be applied to such tasks as the computation of mutual information becomes intractable.
3. For me, it was easy to get lost in many different metrics and keeping in mind what we want to maximize and what we want to minimize for the optimal solutions.

questions:
My questions are based on the weaknesses

1. Could authors please elaborate on weaknesses 1 and 2?
2. According to weakness 3, I would recommend authors to add some additional information to the tables and plots about whether the metric is expected to be minimized or maximized.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700404132

thread_type:
official_review

<thread_object>
creator: reviewer_5hry

<content>
summary:
This paper studies whether actors and critics benefit from a decoupled representation. The authors find that when decoupled, actors and critics learn representations that specialize in different types of information from the environment. Additionally, there is an interplay between these representations even when they are decoupled. The authors show how this insight helps select representation learning objectives for better performance.

strengths:
**originality**
- although the fact that decoupled actor and critic learn different representations is not really surprising, the other findings in the paper are very interesting and novel. 
-  the four mutual information metrics are interesting and helps to understand the effect of decoupled learning. 
- the paper gives some very novel and interesting analysis and results

**quality**
- it is good that authors plan to release all data and code for better reproducibility. 
- overall great quality

**clarity**
- writing is clear and easy to follow. 

**significance**
- empirical and theoretical results are good contributions 
- the analysis and insights from the paper are especially interesting

weaknesses:
I don't have major concerns, minor ones:
- I think it can be nice to explicitly highlight, either in the beginning or conclusion, what are the best representation learning objectives and most important specialisations with quantitative measures. 
- The paper can be even stronger if similar analysis can be done for Q-learning based methods such as DDPG/SAC/TD3 type algorithms. But I understand that might be too much extra work.

questions:
- For the mutual information metrics, e.g., Figure 4, how much will they change if we simply change some of the hyperparameters of one algorithm? I am curious about factors other than coupled/decoupled learning and how much impact they can have on these metrics.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730914852819

thread_type:
official_review

<thread_object>
creator: reviewer_qthx

<content>
summary:
This paper analyzes how different choices of architecture and representation learning objectives affect overfitting on, transfer to, and specializing in different levels of games (ProcGen). The analysis is largely established on the mutual information between different quantities. The author concludes that actor and critic representation specialize in different information, that critic can influence exploration significantly, and that representation learning objectives (i.e., auxiliary loss) should be consistent with the specialization of actor and critic representation.

strengths:
The analysis to representation learning based on mutual information is interesting. The conclusion drawn from this analysis is helpful for deep RL researchers on designing representation learning objectives.

weaknesses:
Writing is very unclear. See my questions for details. Overall, there are overwhelming notations, making the presentation unnecessarily complex. The reason of defining each notation is also not clear to me. Takeaway messages in each section is also not clear. For example, in section 4.1, the author starts with lots of new definitions without explaining why these are necessary. I would suggest the author explain why we need those definitions and then say the high-level messages before diving into details.

questions:
- Line 154: Why measuring how easy it is to infer the identity of the current level from Z is a good measure of overfitting?
- Line 158: The purpose of Theorem 3.1 is not explained. Also, is it a result from the other paper or proved first time in this paper?
- Line 165: Why does a high I(Z_C, V) help optimizing the value loss? 
- Line 166: What do you mean by increasing I(Z_A, V) and what's the relationship between value distillation and increasing I(Z_A, V)?
- Line 175: What do you mean by "whether it is possible to identify latents (ϕ(o), ϕ(o′))) obtained from consecutive observations from pairs stemming from non-consecutive observations?" Is it consecutive or not?
- Line 194: What's the role of linear probing here?
- Equation 6: What are $z^*_{A_0}$ and  $z^*_{A_1}$? Why I((Z^*_A, Z^*'_A); A) is guaranteed to be maximized when following the optimal policy?
- Line 256: I don't see why zero-shot transfer is evident in your example.
- Figure 3: How should one interpret these values? The higher the better? The lower the better? Better summarize the takeaway in the caption.
- Line 374: Why does maximizing J^\pi bias \pi to collect trajectories with high I(O; V)?
- Line 431: Why does actor objective promote learning invariant quantity?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730920294403

thread_type:
official_review

<thread_object>
creator: reviewer_fkit

<content>
summary:
This paper focus on the difference in representation graph between actor and critic in AC methods in RL. The authors aims to more formally validate an empirical finding where decoupling actor and critics may help with representation learning through

strengths:
- The way in which the topic is approached in the section is well considered, I find it readable
- I appreciate the clear problem formulation and motivation of the paper
- The results, while not counter-intuitive nor surprising, gives some form of closure to the decoupling findings in earlier works, and brings this issue to attention does help clarify the topic
- Assumption seems reasonable

weaknesses:
- Potentially need more ground work setup, such as in the problem definition where an observation function is present, which normally implies that the MI between state and observation is not maximized, but it seems the rest of the paper operate on the assumption that it is indeed maximized and so that the environments are fully observable.
- Evaluation is lacking in breath, it would be great to include a general set of tasks.

questions:
- Why use the idea of context, if I'm not mistaken, the entire work can be done with a normal MDP and MDPs with contexts can be seen as a encompassing MDP with different starting states.
- The increased compute and parameter size of decoupled approach may also play a role in learning, is there any intuition or validation on the topic?
- I disagree that "an optimal or near-optimal representation can never be reached under a shared architecture"; from the reduced MDP point of view, I agree that a shared architecture may not be as concise, but still can be optimal (for both policy and value) given that it is a shared architecture.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731188983326

thread_type:
official_review

<thread_object>
creator: reviewer_cooi

<content>
summary:
The paper investigates representation learning objectives for actor and critic networks in deep reinforcement learning, specifically in the context of policy gradient algorithms. Using information-theoretic metrics, the study reveals that the actor and critic networks naturally specialize in capturing distinct information types, with the actor focusing on action-relevant features and the critic on value and environmental dynamics. This specialization is shown to enhance model architecture design and improve representation learning objectives, ultimately contributing to greater sample efficiency in reinforcement learning. In addition, this paper also how various representation learning objectives could help shaping the actor and critic's representation and improve the RL sample efficiency.

strengths:
-  **Novel Perspective on Representation Specialization**: This paper provides a fresh and thorough analysis of actor and critic representation specialization using information-theoretic metrics. 
- **Convincing Empirical Evidence**: The authors provide substantial empirical support for their claims, with extensive experiments that effectively demonstrate the impact of decoupling actor and critic representations, as well as impacts of different representation learning objectives on the resulting representation.
- **Insightful Conclusions**: The study draws interesting and impactful conclusions regarding how specialized representation learning objectives can improve sample efficiency and generalization in RL, a great contribution to the RL community, shedding light on the design of future deep RL algorithms.

weaknesses:
- **Limited Scope of Algorithm Selection**: The paper exclusively focuses on online, on-policy, policy gradient algorithms (e.g., PPO, DCPG), leaving it unclear how well the conclusions apply to other RL frameworks, particularly off-policy algorithms like DDPG or SAC. Broader consideration of diverse algorithm types would strengthen the generalizability of the findings. Also, whether the same conclusion holds under different model sizes? It would be useful to add in the paper to show that this conclusion would still hold if we simply added capacity to the actor/critic network without changing any learning objectives.
- **Presentation and Clarity**: The paper’s structure and presentation could be significantly improved to make it more easier to understand for readers. The writing is currently dense, making it challenging to grasp the main takeaways and hypotheses. For instance, restructuring sections with clearer headings, especially in Sections 6.1 and 6.2, would help readers navigate the study’s conclusions. Using bullet points to highlight key hypotheses and findings, as well as starting each paragraph with a bolded summary sentence, could enhance readability and ensure that readers capture the core messages more effectively.

questions:
- What does the delta percentage in Figure 1 represent? It would be helpful if the authors added a brief explanation to the caption.
- How is mutual information measured in this study? Currently, it’s detailed only in the appendix, but a high-level explanation in the background section would help readers understand the methodology better.
- Could the authors add a performance plot to the figures in the main paper? This would clarify how the information-theoretic metrics relate to final performance outcomes.
- Is it true that $I((Z,Z'); A)$ + $I(Z,Z')$ in general is a good indicator of the algorithm's performance? (Looks like it's the case from Figure 3 and Figure 7.) The authors seem to argue that while actor representation (decoupled) tends to have larger $I((Z,Z'); A)$ while smaller $I(Z,Z')$ and the reverse holds for critic representation, combing them in a coupled representation does not combine the best of both world and resulting in smaller $I((Z,Z'); A)$ + $I(Z,Z')$ and thus worse performance?
- Could Figures 7 and 8 be moved from the appendix into the main paper? The authors might need to shorten other parts of the text rather than placing key results in the appendix while referring back to them in the main sections.
- In Section 6.1, it would be clearer if the authors defined the dynamics prediction objective from Moon et al. with a brief equation or sentence rather than just citing the source. Similarly, in Section 6.2, could the authors introduce the MICo objective with an equation first?
- In Section 6.1, lines 466–468, the distinction between explanations (1) and (2) is unclear; both seem to attribute reduced overfitting to the larger batch size used in value distillation. If the authors hypothesize that batch size is the main factor here, could they test this by running PPO with larger batch sizes, then measuring the relevant metrics to assess any changes in overfitting?
- A broader concern is how the study's conclusions about decoupled versus coupled objectives hold across varying model sizes. For instance, in Section 6.2, the authors note that adding a dynamic prediction objective to the critic impacts certain metrics but harms performance due to an unintended specialization. Similarly, the study attributes overfitting to the coupled representation. Could these conclusions change with larger model capacity (and also training with larger batch sizes as the author claims that this results in more overfitting)? Intuitively, if the shared representation had sufficient capacity (and were trained on larger batches as discussed), could a shared representation still achieve strong performance across the information-theoretic metrics, potentially improving performance without conflicting specializations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732327772027

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Proposed changes to manuscript - part 1: presentation and clarity improvements

comment:
We thank all of the reviewers for their feedback and for spending their time on suggestions that would improve the current manuscript. We will outline here the improvements we have made and plan to make for the new version of the manuscript. We believe these changes address the vast majority of the reviewers' comments and concerns. Following our summary of the reviewers' observations and our responses, we reply directly to each reviewer’s specific questions and comments.
 
**PRESENTATION & CLARITY IMPROVEMENTS**

Several reviewers noted that the presentation and clarity of the paper can be improved. We agree with their assessment. We will list the main issues we have identified from the reviews, and how we are planning to address each.

**Issue n1 - Key takeaways from the study should be stated more clearly.** The reviewers pointed out that it can be challenging to identify the key takeaways in certain sections, and in particular in Section 6.

Broadly, there are four key takeaways from this study:
- The actor and the critic have different optimal representations (covered in Section 4.1)
- Once decoupled, the actor and critic specialise in extracting different information from observations. In practice, their specialisation is broadly consistent with what we expect from their respective optimal representation. (covered in Section 4.2)
- For the actor, representation learning objectives are effective when they promote capturing information shared across all training levels, either explicitly or implicitly. (covered in Section 6.1)
- The critic is capable of influencing exploration to benefit its own learning objective (covered in Section 5), and auxiliary objective(s) (covered in Section 6.2).

To ensure the above takeaways are clearly communicated, we will:
- Rework the structure of sections 5 and 6 so that each section focuses on a single takeaway.
- Ensure these takeaways get prominently featured within their respective section, but also in the introduction, abstract and conclusion. 
- Rework certain figures (for example figure 1) to improve how they communicate this information.


**Issue n2 - It is difficult to keep track of what each of the four mutual information (MI) metrics measures and what they say about the representation.** We believe this problem stems from the reader not being introduced to the different MI metrics until Section 3. Furthermore, the section is long and contains a lot of mathematical notation, making it difficult to refer back to.

To address this issue, we will rework the table in Figure 1 (right) as a standalone Table containing compact and informal definitions for each MI metrics and what they quantify. The new version can be seen at https://ibb.co/vwHnfxS

In doing so, we aim to familiarise the reader with the MI metrics as early as the introduction and to have the new Table 1 be a "cheatsheet" to refer to when going through the rest of the paper.
In addition, we will make a range of minor improvements to improve clarity further, outlined in individual comments to the reviewers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732794624651

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
New and improved version of the manuscript

comment:
We thank the reviewers again for their helpful feedback which improved the paper significantly. Please see the new version of the manuscript, which includes the reviewers' suggestions for experimentation with the number of learned parameters, experimentation in continuous control environments, and experimentation with batch sizes. We have also included more experiments with additional auxiliary objectives and base algorithm combinations. Due to the large number of algorithms studied in this work, many plots are half the length of an entire page and, therefore, must be delegated to the appendix. However, we ensured that the paper's critical points are supported by plots in the main body of the paper by extracting specific elements from the full plots in the appendix.

Finally, we have improved the paper's writing and presentation clarity. For example, we now include a table at the beginning of the paper that clearly defines each of the mutual information measures we consider and gives an example of what each quantity captures.

If selected for publication, we plan to further improve the paper by including even more detailed information in the appendix, such as the training curves for each algorithm in each of the environments tested, detailed descriptions (in addition to the citations) for each representation learning objective we study, and more.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper studies the representation of actor and critic in on-policy RL algorithms using information theory. Overall, the reviewers are positive about the paper. Multiple reviewers raises concerns on clarity during the discussion which I think is mostly addressed by the authors' response. Some reviewer mention that it's unclear how the current paper's finding may apply to off-policy algorithms. I think this is a serious concern. The author should clearly highlight the scope of algorithms that are analyzed here e.g. in title, abstract, and introduction (e.g. main finding summaries). The current writing (e.g. title and abstract) suggests a very broad over claim for all actor critic algorithms, which can be misleading. 

**Please fix these writing issues and clarify the scope the findings are this paper is applicable to.**